2024-09-16T12:09:36.772 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T12:09:36.793 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T12:09:36.880 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907248 branch: wip-bharath9-testing-2024-09-13-1444-quincy description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/compare-mirror-images-nbd} email: sangadi@redhat.com first_in_suite: false job_id: '7907248' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each name: sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-bharath9-testing-2024-09-13-1444-quincy ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.4: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.5: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.6: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluefs allocator: hybrid bluestore allocator: hybrid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: d75f23820806da622f561ee5ee7551015285993f ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: extra_packages: - rbd-nbd flavor: default sha1: d75f23820806da622f561ee5ee7551015285993f extra_system_packages: - pv thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath9-testing-2024-09-13-1444-quincy sha1: d75f23820806da622f561ee5ee7551015285993f owner: scheduled_sangadi@teuthology priority: 130 repo: https://github.com/ceph/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 9764 sha1: d75f23820806da622f561ee5ee7551015285993f sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: wip-bharath9-testing-2024-09-13-1444-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d75f23820806da622f561ee5ee7551015285993f targets: smithi018.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd' - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - workunit: clients: cluster1.client.mirror: - rbd/compare_mirror_images.sh env: RBD_DEVICE_TYPE: nbd timeout: 3h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-16_10:44:09 tube: smithi user: sangadi verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T12:09:36.880 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa; will attempt to use it 2024-09-16T12:09:36.881 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa/tasks 2024-09-16T12:09:36.881 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T12:09:36.883 INFO:teuthology.task.internal:Checking packages... 2024-09-16T12:09:36.907 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'd75f23820806da622f561ee5ee7551015285993f' 2024-09-16T12:09:36.907 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T12:09:36.907 INFO:teuthology.packaging:ref: None 2024-09-16T12:09:36.907 INFO:teuthology.packaging:tag: None 2024-09-16T12:09:36.908 INFO:teuthology.packaging:branch: wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-16T12:09:36.908 INFO:teuthology.packaging:sha1: d75f23820806da622f561ee5ee7551015285993f 2024-09-16T12:09:36.908 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-16T12:09:37.051 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1643-gd75f2382-1focal 2024-09-16T12:09:37.053 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T12:09:37.059 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T12:09:37.059 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T12:09:37.189 INFO:teuthology.task.internal:Saving configuration 2024-09-16T12:09:37.206 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T12:09:37.282 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T12:09:37.307 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi018.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907248', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 12:03:35.752793', 'locked_by': 'scheduled_sangadi@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-16T12:09:37.329 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907248', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 12:03:35.751619', 'locked_by': 'scheduled_sangadi@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-16T12:09:37.329 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T12:09:37.336 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-09-16T12:09:37.336 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-09-16T12:09:37.336 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T12:09:37.398 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa14b8c3ac0>, signals=[15]) 2024-09-16T12:09:37.398 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T12:09:37.406 INFO:teuthology.task.internal:Opening connections... 2024-09-16T12:09:37.406 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi018.front.sepia.ceph.com 2024-09-16T12:09:37.407 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T12:09:37.504 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2024-09-16T12:09:37.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T12:09:37.591 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T12:09:37.598 DEBUG:teuthology.orchestra.run.smithi018:> uname -m 2024-09-16T12:09:37.605 INFO:teuthology.orchestra.run.smithi018.stdout:x86_64 2024-09-16T12:09:37.606 DEBUG:teuthology.orchestra.run.smithi018:> cat /etc/os-release 2024-09-16T12:09:37.655 INFO:teuthology.orchestra.run.smithi018.stdout:NAME="Ubuntu" 2024-09-16T12:09:37.656 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-16T12:09:37.656 INFO:teuthology.orchestra.run.smithi018.stdout:ID=ubuntu 2024-09-16T12:09:37.656 INFO:teuthology.orchestra.run.smithi018.stdout:ID_LIKE=debian 2024-09-16T12:09:37.656 INFO:teuthology.orchestra.run.smithi018.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-16T12:09:37.656 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION_ID="20.04" 2024-09-16T12:09:37.656 INFO:teuthology.orchestra.run.smithi018.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T12:09:37.656 INFO:teuthology.orchestra.run.smithi018.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T12:09:37.656 INFO:teuthology.orchestra.run.smithi018.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T12:09:37.656 INFO:teuthology.orchestra.run.smithi018.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T12:09:37.656 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION_CODENAME=focal 2024-09-16T12:09:37.656 INFO:teuthology.orchestra.run.smithi018.stdout:UBUNTU_CODENAME=focal 2024-09-16T12:09:37.657 INFO:teuthology.lock.ops:Updating smithi018.front.sepia.ceph.com on lock server 2024-09-16T12:09:37.678 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2024-09-16T12:09:37.684 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2024-09-16T12:09:37.685 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2024-09-16T12:09:37.733 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="Ubuntu" 2024-09-16T12:09:37.733 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-16T12:09:37.734 INFO:teuthology.orchestra.run.smithi172.stdout:ID=ubuntu 2024-09-16T12:09:37.734 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE=debian 2024-09-16T12:09:37.734 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-16T12:09:37.734 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="20.04" 2024-09-16T12:09:37.734 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T12:09:37.734 INFO:teuthology.orchestra.run.smithi172.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T12:09:37.734 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T12:09:37.734 INFO:teuthology.orchestra.run.smithi172.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T12:09:37.734 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_CODENAME=focal 2024-09-16T12:09:37.734 INFO:teuthology.orchestra.run.smithi172.stdout:UBUNTU_CODENAME=focal 2024-09-16T12:09:37.735 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2024-09-16T12:09:37.753 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T12:09:37.763 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T12:09:37.816 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T12:09:37.816 DEBUG:teuthology.orchestra.run.smithi018:> test '!' -e /home/ubuntu/cephtest 2024-09-16T12:09:37.818 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2024-09-16T12:09:37.823 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T12:09:37.875 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T12:09:37.876 DEBUG:teuthology.orchestra.run.smithi018:> test -z $(ls -A /var/lib/ceph) 2024-09-16T12:09:37.878 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2024-09-16T12:09:37.911 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T12:09:37.994 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T12:09:38.035 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T12:09:38.035 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T12:09:38.035 DEBUG:teuthology.orchestra.run.smithi018:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T12:09:38.036 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T12:09:38.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:09:38.042 DEBUG:teuthology.orchestra.run.smithi018:> uname -r 2024-09-16T12:09:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:09:38.043 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-09-16T12:09:38.049 INFO:teuthology.orchestra.run.smithi172.stdout:5.4.0-124-generic 2024-09-16T12:09:38.049 INFO:teuthology.task.kernel:Running kernel on smithi172: 5.4.0-124-generic 2024-09-16T12:09:38.049 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get clean 2024-09-16T12:09:38.091 INFO:teuthology.orchestra.run.smithi018.stdout:5.4.0-124-generic 2024-09-16T12:09:38.091 INFO:teuthology.task.kernel:Running kernel on smithi018: 5.4.0-124-generic 2024-09-16T12:09:38.091 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt-get clean 2024-09-16T12:09:38.159 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2024-09-16T12:09:38.209 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt-get update 2024-09-16T12:09:38.328 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T12:09:38.338 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-16T12:09:38.367 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T12:09:38.385 INFO:teuthology.orchestra.run.smithi018.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-16T12:09:38.401 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-16T12:09:38.435 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-16T12:09:38.449 INFO:teuthology.orchestra.run.smithi018.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-16T12:09:38.482 INFO:teuthology.orchestra.run.smithi018.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-16T12:09:38.715 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,563 kB] 2024-09-16T12:09:38.744 INFO:teuthology.orchestra.run.smithi018.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-16T12:09:38.831 INFO:teuthology.orchestra.run.smithi018.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,563 kB] 2024-09-16T12:09:38.844 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-16T12:09:38.856 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-16T12:09:38.863 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-16T12:09:38.863 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,236 kB] 2024-09-16T12:09:38.871 INFO:teuthology.orchestra.run.smithi018.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-16T12:09:38.875 INFO:teuthology.orchestra.run.smithi018.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-16T12:09:38.875 INFO:teuthology.orchestra.run.smithi018.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,236 kB] 2024-09-16T12:09:38.898 INFO:teuthology.orchestra.run.smithi018.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-16T12:09:38.898 INFO:teuthology.orchestra.run.smithi018.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [453 kB] 2024-09-16T12:09:38.901 INFO:teuthology.orchestra.run.smithi172.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-16T12:09:38.902 INFO:teuthology.orchestra.run.smithi018.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T12:09:38.902 INFO:teuthology.orchestra.run.smithi018.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-16T12:09:38.902 INFO:teuthology.orchestra.run.smithi172.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [453 kB] 2024-09-16T12:09:38.907 INFO:teuthology.orchestra.run.smithi172.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T12:09:38.907 INFO:teuthology.orchestra.run.smithi172.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-16T12:09:38.907 INFO:teuthology.orchestra.run.smithi018.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,230 kB] 2024-09-16T12:09:38.912 INFO:teuthology.orchestra.run.smithi172.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,230 kB] 2024-09-16T12:09:38.916 INFO:teuthology.orchestra.run.smithi018.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-16T12:09:38.919 INFO:teuthology.orchestra.run.smithi018.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-16T12:09:38.919 INFO:teuthology.orchestra.run.smithi018.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-16T12:09:38.919 INFO:teuthology.orchestra.run.smithi018.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-16T12:09:38.920 INFO:teuthology.orchestra.run.smithi172.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-16T12:09:38.922 INFO:teuthology.orchestra.run.smithi172.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-16T12:09:38.922 INFO:teuthology.orchestra.run.smithi172.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-16T12:09:38.923 INFO:teuthology.orchestra.run.smithi172.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-16T12:09:38.923 INFO:teuthology.orchestra.run.smithi172.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-16T12:09:38.923 INFO:teuthology.orchestra.run.smithi172.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-16T12:09:38.923 INFO:teuthology.orchestra.run.smithi018.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-16T12:09:38.924 INFO:teuthology.orchestra.run.smithi018.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-16T12:09:38.924 INFO:teuthology.orchestra.run.smithi018.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-16T12:09:38.925 INFO:teuthology.orchestra.run.smithi018.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-16T12:09:38.937 INFO:teuthology.orchestra.run.smithi172.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-16T12:09:38.938 INFO:teuthology.orchestra.run.smithi172.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-16T12:09:38.938 INFO:teuthology.orchestra.run.smithi172.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-16T12:09:38.941 INFO:teuthology.orchestra.run.smithi018.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-16T12:09:38.941 INFO:teuthology.orchestra.run.smithi018.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-16T12:09:38.942 INFO:teuthology.orchestra.run.smithi018.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-16T12:09:38.942 INFO:teuthology.orchestra.run.smithi018.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-16T12:09:38.957 INFO:teuthology.orchestra.run.smithi172.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-16T12:09:38.957 INFO:teuthology.orchestra.run.smithi172.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-16T12:09:38.958 INFO:teuthology.orchestra.run.smithi172.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-16T12:09:39.039 INFO:teuthology.orchestra.run.smithi172.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-16T12:09:39.072 INFO:teuthology.orchestra.run.smithi018.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-16T12:09:39.558 INFO:teuthology.orchestra.run.smithi018.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-16T12:09:39.572 INFO:teuthology.orchestra.run.smithi018.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-16T12:09:39.581 INFO:teuthology.orchestra.run.smithi018.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-16T12:09:39.582 INFO:teuthology.orchestra.run.smithi018.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-16T12:09:39.582 INFO:teuthology.orchestra.run.smithi018.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-16T12:09:39.583 INFO:teuthology.orchestra.run.smithi172.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-16T12:09:39.597 INFO:teuthology.orchestra.run.smithi172.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-16T12:09:39.605 INFO:teuthology.orchestra.run.smithi172.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-16T12:09:39.605 INFO:teuthology.orchestra.run.smithi172.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-16T12:09:39.605 INFO:teuthology.orchestra.run.smithi172.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-16T12:09:39.670 INFO:teuthology.orchestra.run.smithi018.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-16T12:09:39.677 INFO:teuthology.orchestra.run.smithi018.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T12:09:39.678 INFO:teuthology.orchestra.run.smithi018.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-16T12:09:39.691 INFO:teuthology.orchestra.run.smithi018.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-16T12:09:39.695 INFO:teuthology.orchestra.run.smithi172.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-16T12:09:39.701 INFO:teuthology.orchestra.run.smithi172.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-16T12:09:39.701 INFO:teuthology.orchestra.run.smithi172.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-16T12:09:39.739 INFO:teuthology.orchestra.run.smithi018.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-16T12:09:39.743 INFO:teuthology.orchestra.run.smithi018.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-16T12:09:39.743 INFO:teuthology.orchestra.run.smithi018.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-16T12:09:39.743 INFO:teuthology.orchestra.run.smithi018.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-16T12:09:39.743 INFO:teuthology.orchestra.run.smithi018.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-16T12:09:39.752 INFO:teuthology.orchestra.run.smithi172.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-16T12:09:39.764 INFO:teuthology.orchestra.run.smithi172.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-16T12:09:39.768 INFO:teuthology.orchestra.run.smithi172.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-16T12:09:39.769 INFO:teuthology.orchestra.run.smithi172.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-16T12:09:39.772 INFO:teuthology.orchestra.run.smithi172.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-16T12:09:39.772 INFO:teuthology.orchestra.run.smithi172.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-16T12:09:39.819 INFO:teuthology.orchestra.run.smithi018.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-16T12:09:39.841 INFO:teuthology.orchestra.run.smithi172.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-16T12:09:43.354 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 21.8 MB in 3s (7,123 kB/s) 2024-09-16T12:09:43.451 INFO:teuthology.orchestra.run.smithi018.stdout:Fetched 21.8 MB in 3s (7,009 kB/s) 2024-09-16T12:09:44.957 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-09-16T12:09:44.983 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T12:09:45.033 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2024-09-16T12:09:45.056 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-09-16T12:09:45.061 DEBUG:teuthology.orchestra.run.smithi018:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T12:09:45.139 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2024-09-16T12:09:45.275 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2024-09-16T12:09:45.275 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2024-09-16T12:09:45.367 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2024-09-16T12:09:45.368 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2024-09-16T12:09:45.480 INFO:teuthology.orchestra.run.smithi172.stdout:The following additional packages will be installed: 2024-09-16T12:09:45.480 INFO:teuthology.orchestra.run.smithi172.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T12:09:45.480 INFO:teuthology.orchestra.run.smithi172.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-16T12:09:45.480 INFO:teuthology.orchestra.run.smithi172.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-16T12:09:45.481 INFO:teuthology.orchestra.run.smithi172.stdout:Suggested packages: 2024-09-16T12:09:45.481 INFO:teuthology.orchestra.run.smithi172.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-16T12:09:45.481 INFO:teuthology.orchestra.run.smithi172.stdout:Recommended packages: 2024-09-16T12:09:45.481 INFO:teuthology.orchestra.run.smithi172.stdout: thermald 2024-09-16T12:09:45.547 INFO:teuthology.orchestra.run.smithi172.stdout:The following NEW packages will be installed: 2024-09-16T12:09:45.547 INFO:teuthology.orchestra.run.smithi172.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T12:09:45.547 INFO:teuthology.orchestra.run.smithi172.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-16T12:09:45.547 INFO:teuthology.orchestra.run.smithi172.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-16T12:09:45.549 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages will be upgraded: 2024-09-16T12:09:45.549 INFO:teuthology.orchestra.run.smithi172.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T12:09:45.586 INFO:teuthology.orchestra.run.smithi018.stdout:The following additional packages will be installed: 2024-09-16T12:09:45.586 INFO:teuthology.orchestra.run.smithi018.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T12:09:45.587 INFO:teuthology.orchestra.run.smithi018.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-16T12:09:45.587 INFO:teuthology.orchestra.run.smithi018.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-16T12:09:45.588 INFO:teuthology.orchestra.run.smithi018.stdout:Suggested packages: 2024-09-16T12:09:45.588 INFO:teuthology.orchestra.run.smithi018.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-16T12:09:45.588 INFO:teuthology.orchestra.run.smithi018.stdout:Recommended packages: 2024-09-16T12:09:45.588 INFO:teuthology.orchestra.run.smithi018.stdout: thermald 2024-09-16T12:09:45.602 INFO:teuthology.orchestra.run.smithi172.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T12:09:45.603 INFO:teuthology.orchestra.run.smithi172.stdout:Need to get 77.2 MB of archives. 2024-09-16T12:09:45.603 INFO:teuthology.orchestra.run.smithi172.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-16T12:09:45.603 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-16T12:09:45.642 INFO:teuthology.orchestra.run.smithi018.stdout:The following NEW packages will be installed: 2024-09-16T12:09:45.642 INFO:teuthology.orchestra.run.smithi018.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-16T12:09:45.642 INFO:teuthology.orchestra.run.smithi018.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-16T12:09:45.643 INFO:teuthology.orchestra.run.smithi018.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-16T12:09:45.644 INFO:teuthology.orchestra.run.smithi018.stdout:The following packages will be upgraded: 2024-09-16T12:09:45.645 INFO:teuthology.orchestra.run.smithi018.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T12:09:45.699 INFO:teuthology.orchestra.run.smithi018.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T12:09:45.699 INFO:teuthology.orchestra.run.smithi018.stdout:Need to get 77.2 MB of archives. 2024-09-16T12:09:45.699 INFO:teuthology.orchestra.run.smithi018.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-16T12:09:45.699 INFO:teuthology.orchestra.run.smithi018.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-16T12:09:45.928 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-16T12:09:46.019 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-16T12:09:46.032 INFO:teuthology.orchestra.run.smithi018.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-16T12:09:46.127 INFO:teuthology.orchestra.run.smithi018.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-16T12:09:46.483 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-16T12:09:46.483 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-16T12:09:46.483 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-16T12:09:46.577 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-16T12:09:46.587 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-16T12:09:46.630 INFO:teuthology.orchestra.run.smithi018.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-16T12:09:46.631 INFO:teuthology.orchestra.run.smithi018.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-16T12:09:46.631 INFO:teuthology.orchestra.run.smithi018.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-16T12:09:46.726 INFO:teuthology.orchestra.run.smithi018.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-16T12:09:46.736 INFO:teuthology.orchestra.run.smithi018.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-16T12:09:47.263 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 77.2 MB in 1s (74.9 MB/s) 2024-09-16T12:09:47.443 INFO:teuthology.orchestra.run.smithi018.stdout:Fetched 77.2 MB in 1s (71.2 MB/s) 2024-09-16T12:09:47.478 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-16T12:09:47.622 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-16T12:09:49.211 INFO:teuthology.orchestra.run.smithi172.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-16T12:09:49.216 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T12:09:49.320 INFO:teuthology.orchestra.run.smithi018.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-16T12:09:49.325 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T12:09:49.359 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T12:09:49.552 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T12:09:51.368 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-16T12:09:51.384 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T12:09:51.419 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-16T12:09:51.435 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T12:09:51.482 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T12:09:51.535 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T12:09:52.587 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-16T12:09:52.601 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T12:09:52.664 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-16T12:09:52.670 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T12:09:52.678 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T12:09:52.722 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T12:09:57.475 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T12:09:57.598 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T12:09:57.601 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T12:09:57.749 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T12:09:57.977 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T12:09:58.083 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T12:09:58.105 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T12:09:58.228 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T12:09:58.466 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-16T12:09:58.483 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-16T12:09:58.541 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T12:09:58.564 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-16T12:09:58.581 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-16T12:09:58.622 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T12:10:03.957 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-16T12:10:03.979 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T12:10:04.024 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T12:10:04.424 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-16T12:10:04.446 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-16T12:10:04.490 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T12:10:06.907 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T12:10:07.060 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T12:10:07.354 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-16T12:10:07.390 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T12:10:07.484 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-16T12:10:07.851 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T12:10:08.682 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T12:10:09.115 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T12:10:09.875 INFO:teuthology.orchestra.run.smithi018.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-16T12:10:09.875 INFO:teuthology.orchestra.run.smithi018.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-16T12:10:10.009 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T12:10:10.275 INFO:teuthology.orchestra.run.smithi172.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-16T12:10:10.275 INFO:teuthology.orchestra.run.smithi172.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-16T12:10:10.433 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T12:10:11.236 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T12:10:11.396 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-16T12:10:11.505 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T12:10:11.631 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-16T12:10:11.757 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-16T12:10:11.900 INFO:teuthology.orchestra.run.smithi018.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T12:10:11.971 INFO:teuthology.orchestra.run.smithi018.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T12:10:11.971 INFO:teuthology.orchestra.run.smithi018.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-16T12:10:12.103 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-16T12:10:12.228 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-16T12:10:12.398 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T12:10:12.522 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-16T12:10:12.648 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-16T12:10:12.791 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-16T12:10:12.865 INFO:teuthology.orchestra.run.smithi172.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T12:10:12.865 INFO:teuthology.orchestra.run.smithi172.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-16T12:10:34.652 INFO:teuthology.orchestra.run.smithi172.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T12:10:34.652 INFO:teuthology.orchestra.run.smithi172.stdout:Sourcing file `/etc/default/grub' 2024-09-16T12:10:34.673 INFO:teuthology.orchestra.run.smithi172.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T12:10:34.700 INFO:teuthology.orchestra.run.smithi172.stdout:Generating grub configuration file ... 2024-09-16T12:10:34.969 INFO:teuthology.orchestra.run.smithi018.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T12:10:34.969 INFO:teuthology.orchestra.run.smithi018.stdout:Sourcing file `/etc/default/grub' 2024-09-16T12:10:34.977 INFO:teuthology.orchestra.run.smithi018.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T12:10:34.998 INFO:teuthology.orchestra.run.smithi018.stdout:Generating grub configuration file ... 2024-09-16T12:10:35.069 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T12:10:35.090 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T12:10:35.317 INFO:teuthology.orchestra.run.smithi018.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T12:10:35.336 INFO:teuthology.orchestra.run.smithi018.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T12:10:35.505 INFO:teuthology.orchestra.run.smithi172.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T12:10:35.516 INFO:teuthology.orchestra.run.smithi172.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T12:10:35.735 INFO:teuthology.orchestra.run.smithi018.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T12:10:35.747 INFO:teuthology.orchestra.run.smithi018.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T12:10:37.634 INFO:teuthology.orchestra.run.smithi172.stdout:done 2024-09-16T12:10:37.772 INFO:teuthology.orchestra.run.smithi018.stdout:done 2024-09-16T12:10:38.048 DEBUG:teuthology.orchestra.run.smithi172:> dpkg -s linux-image-generic 2024-09-16T12:10:38.094 INFO:teuthology.orchestra.run.smithi172.stdout:Package: linux-image-generic 2024-09-16T12:10:38.094 INFO:teuthology.orchestra.run.smithi172.stdout:Status: install ok installed 2024-09-16T12:10:38.094 INFO:teuthology.orchestra.run.smithi172.stdout:Priority: optional 2024-09-16T12:10:38.095 INFO:teuthology.orchestra.run.smithi172.stdout:Section: kernel 2024-09-16T12:10:38.095 INFO:teuthology.orchestra.run.smithi172.stdout:Installed-Size: 20 2024-09-16T12:10:38.095 INFO:teuthology.orchestra.run.smithi172.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T12:10:38.095 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: amd64 2024-09-16T12:10:38.095 INFO:teuthology.orchestra.run.smithi172.stdout:Source: linux-meta 2024-09-16T12:10:38.095 INFO:teuthology.orchestra.run.smithi172.stdout:Version: 5.4.0.195.193 2024-09-16T12:10:38.095 INFO:teuthology.orchestra.run.smithi172.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T12:10:38.095 INFO:teuthology.orchestra.run.smithi172.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T12:10:38.095 INFO:teuthology.orchestra.run.smithi172.stdout:Recommends: thermald 2024-09-16T12:10:38.095 INFO:teuthology.orchestra.run.smithi172.stdout:Description: Generic Linux kernel image 2024-09-16T12:10:38.095 INFO:teuthology.orchestra.run.smithi172.stdout: This package will always depend on the latest generic kernel image 2024-09-16T12:10:38.095 INFO:teuthology.orchestra.run.smithi172.stdout: available. 2024-09-16T12:10:38.096 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-16T12:10:38.096 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T12:10:38.096 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T12:10:38.096 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-09-16T12:10:38.096 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi172.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T12:10:38.096 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get clean 2024-09-16T12:10:38.169 DEBUG:teuthology.orchestra.run.smithi018:> dpkg -s linux-image-generic 2024-09-16T12:10:38.180 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2024-09-16T12:10:38.228 INFO:teuthology.orchestra.run.smithi018.stdout:Package: linux-image-generic 2024-09-16T12:10:38.228 INFO:teuthology.orchestra.run.smithi018.stdout:Status: install ok installed 2024-09-16T12:10:38.228 INFO:teuthology.orchestra.run.smithi018.stdout:Priority: optional 2024-09-16T12:10:38.229 INFO:teuthology.orchestra.run.smithi018.stdout:Section: kernel 2024-09-16T12:10:38.229 INFO:teuthology.orchestra.run.smithi018.stdout:Installed-Size: 20 2024-09-16T12:10:38.229 INFO:teuthology.orchestra.run.smithi018.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T12:10:38.229 INFO:teuthology.orchestra.run.smithi018.stdout:Architecture: amd64 2024-09-16T12:10:38.229 INFO:teuthology.orchestra.run.smithi018.stdout:Source: linux-meta 2024-09-16T12:10:38.229 INFO:teuthology.orchestra.run.smithi018.stdout:Version: 5.4.0.195.193 2024-09-16T12:10:38.229 INFO:teuthology.orchestra.run.smithi018.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T12:10:38.229 INFO:teuthology.orchestra.run.smithi018.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T12:10:38.229 INFO:teuthology.orchestra.run.smithi018.stdout:Recommends: thermald 2024-09-16T12:10:38.229 INFO:teuthology.orchestra.run.smithi018.stdout:Description: Generic Linux kernel image 2024-09-16T12:10:38.229 INFO:teuthology.orchestra.run.smithi018.stdout: This package will always depend on the latest generic kernel image 2024-09-16T12:10:38.229 INFO:teuthology.orchestra.run.smithi018.stdout: available. 2024-09-16T12:10:38.230 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-16T12:10:38.230 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T12:10:38.230 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T12:10:38.230 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-09-16T12:10:38.230 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi018.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T12:10:38.230 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt-get clean 2024-09-16T12:10:38.324 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T12:10:38.327 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt-get update 2024-09-16T12:10:38.331 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T12:10:38.332 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:3 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T12:10:38.348 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T12:10:38.471 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T12:10:38.589 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T12:10:38.671 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T12:10:38.752 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T12:10:40.200 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-09-16T12:10:40.226 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T12:10:40.298 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-09-16T12:10:40.361 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2024-09-16T12:10:40.389 DEBUG:teuthology.orchestra.run.smithi018:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T12:10:40.464 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2024-09-16T12:10:40.492 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2024-09-16T12:10:40.492 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2024-09-16T12:10:40.664 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2024-09-16T12:10:40.665 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2024-09-16T12:10:40.725 INFO:teuthology.orchestra.run.smithi172.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-16T12:10:40.726 INFO:teuthology.orchestra.run.smithi172.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T12:10:40.727 DEBUG:teuthology.orchestra.run.smithi172:> dpkg -s linux-image-generic 2024-09-16T12:10:40.748 INFO:teuthology.orchestra.run.smithi172.stdout:Package: linux-image-generic 2024-09-16T12:10:40.748 INFO:teuthology.orchestra.run.smithi172.stdout:Status: install ok installed 2024-09-16T12:10:40.748 INFO:teuthology.orchestra.run.smithi172.stdout:Priority: optional 2024-09-16T12:10:40.748 INFO:teuthology.orchestra.run.smithi172.stdout:Section: kernel 2024-09-16T12:10:40.748 INFO:teuthology.orchestra.run.smithi172.stdout:Installed-Size: 20 2024-09-16T12:10:40.748 INFO:teuthology.orchestra.run.smithi172.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T12:10:40.749 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: amd64 2024-09-16T12:10:40.749 INFO:teuthology.orchestra.run.smithi172.stdout:Source: linux-meta 2024-09-16T12:10:40.749 INFO:teuthology.orchestra.run.smithi172.stdout:Version: 5.4.0.195.193 2024-09-16T12:10:40.749 INFO:teuthology.orchestra.run.smithi172.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T12:10:40.749 INFO:teuthology.orchestra.run.smithi172.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T12:10:40.749 INFO:teuthology.orchestra.run.smithi172.stdout:Recommends: thermald 2024-09-16T12:10:40.749 INFO:teuthology.orchestra.run.smithi172.stdout:Description: Generic Linux kernel image 2024-09-16T12:10:40.749 INFO:teuthology.orchestra.run.smithi172.stdout: This package will always depend on the latest generic kernel image 2024-09-16T12:10:40.749 INFO:teuthology.orchestra.run.smithi172.stdout: available. 2024-09-16T12:10:40.750 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2024-09-16T12:10:40.754 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.srjzRqG8Je 2024-09-16T12:10:40.754 DEBUG:teuthology.orchestra.run.smithi172:> sudo cp /boot/grub/grub.cfg /tmp/tmp.srjzRqG8Je 2024-09-16T12:10:40.809 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.srjzRqG8Je 2024-09-16T12:10:40.845 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.srjzRqG8Je is 9KB 2024-09-16T12:10:40.859 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.srjzRqG8Je 2024-09-16T12:10:40.865 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T12:10:40.907 INFO:teuthology.orchestra.run.smithi018.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-16T12:10:40.907 INFO:teuthology.orchestra.run.smithi018.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-16T12:10:40.909 DEBUG:teuthology.orchestra.run.smithi018:> dpkg -s linux-image-generic 2024-09-16T12:10:40.923 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:10:40.923 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T12:10:40.924 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T12:10:40.927 INFO:teuthology.orchestra.run.smithi018.stdout:Package: linux-image-generic 2024-09-16T12:10:40.927 INFO:teuthology.orchestra.run.smithi018.stdout:Status: install ok installed 2024-09-16T12:10:40.927 INFO:teuthology.orchestra.run.smithi018.stdout:Priority: optional 2024-09-16T12:10:40.927 INFO:teuthology.orchestra.run.smithi018.stdout:Section: kernel 2024-09-16T12:10:40.928 INFO:teuthology.orchestra.run.smithi018.stdout:Installed-Size: 20 2024-09-16T12:10:40.928 INFO:teuthology.orchestra.run.smithi018.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T12:10:40.928 INFO:teuthology.orchestra.run.smithi018.stdout:Architecture: amd64 2024-09-16T12:10:40.928 INFO:teuthology.orchestra.run.smithi018.stdout:Source: linux-meta 2024-09-16T12:10:40.928 INFO:teuthology.orchestra.run.smithi018.stdout:Version: 5.4.0.195.193 2024-09-16T12:10:40.928 INFO:teuthology.orchestra.run.smithi018.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-16T12:10:40.928 INFO:teuthology.orchestra.run.smithi018.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T12:10:40.928 INFO:teuthology.orchestra.run.smithi018.stdout:Recommends: thermald 2024-09-16T12:10:40.928 INFO:teuthology.orchestra.run.smithi018.stdout:Description: Generic Linux kernel image 2024-09-16T12:10:40.928 INFO:teuthology.orchestra.run.smithi018.stdout: This package will always depend on the latest generic kernel image 2024-09-16T12:10:40.928 INFO:teuthology.orchestra.run.smithi018.stdout: available. 2024-09-16T12:10:40.929 DEBUG:teuthology.orchestra.run.smithi018:> mktemp 2024-09-16T12:10:40.935 INFO:teuthology.orchestra.run.smithi018.stdout:/tmp/tmp.A57JIVf3Un 2024-09-16T12:10:40.935 DEBUG:teuthology.orchestra.run.smithi018:> sudo cp /boot/grub/grub.cfg /tmp/tmp.A57JIVf3Un 2024-09-16T12:10:40.945 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0666 /tmp/tmp.A57JIVf3Un 2024-09-16T12:10:40.958 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-16T12:10:40.958 DEBUG:teuthology.orchestra.run.smithi172:> sudo update-grub 2024-09-16T12:10:40.979 DEBUG:teuthology.orchestra.remote:smithi018:/tmp/tmp.A57JIVf3Un is 9KB 2024-09-16T12:10:40.992 DEBUG:teuthology.orchestra.run.smithi018:> rm -fr /tmp/tmp.A57JIVf3Un 2024-09-16T12:10:40.998 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T12:10:41.057 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:10:41.057 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T12:10:41.057 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T12:10:41.086 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-16T12:10:41.086 DEBUG:teuthology.orchestra.run.smithi018:> sudo update-grub 2024-09-16T12:10:42.447 INFO:teuthology.orchestra.run.smithi172.stderr:Sourcing file `/etc/default/grub' 2024-09-16T12:10:42.454 INFO:teuthology.orchestra.run.smithi172.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T12:10:42.465 INFO:teuthology.orchestra.run.smithi018.stderr:Sourcing file `/etc/default/grub' 2024-09-16T12:10:42.465 INFO:teuthology.orchestra.run.smithi172.stderr:Generating grub configuration file ... 2024-09-16T12:10:42.467 INFO:teuthology.orchestra.run.smithi018.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T12:10:42.481 INFO:teuthology.orchestra.run.smithi018.stderr:Generating grub configuration file ... 2024-09-16T12:10:42.714 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T12:10:42.726 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T12:10:42.752 INFO:teuthology.orchestra.run.smithi018.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-16T12:10:42.763 INFO:teuthology.orchestra.run.smithi018.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-16T12:10:43.112 INFO:teuthology.orchestra.run.smithi018.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T12:10:43.116 INFO:teuthology.orchestra.run.smithi172.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-16T12:10:43.124 INFO:teuthology.orchestra.run.smithi018.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T12:10:43.127 INFO:teuthology.orchestra.run.smithi172.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-16T12:10:44.088 INFO:teuthology.orchestra.run.smithi018.stderr:done 2024-09-16T12:10:44.090 DEBUG:teuthology.orchestra.run.smithi018:> sudo shutdown -r now 2024-09-16T12:10:44.103 INFO:teuthology.orchestra.run.smithi172.stderr:done 2024-09-16T12:10:44.105 DEBUG:teuthology.orchestra.run.smithi172:> sudo shutdown -r now 2024-09-16T12:11:14.118 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-09-16T12:11:14.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-09-16T12:11:14.119 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T12:11:14.121 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-09-16T12:11:14.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-09-16T12:11:14.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T12:11:48.669 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-09-16T12:11:57.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-09-16T12:11:57.678 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T12:12:14.137 DEBUG:teuthology.orchestra.remote:timed out 2024-09-16T12:12:16.061 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-09-16T12:12:23.141 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-09-16T12:12:23.142 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T12:12:26.177 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.18 2024-09-16T12:12:28.064 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-09-16T12:12:28.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T12:12:28.291 ERROR:teuthology.orchestra.connection:Error authenticating with smithi172.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/connection.py", line 111, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 485, in connect self._auth( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 818, in _auth raise saved_exception File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 794, in _auth self._transport.auth_publickey(username, key) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1664, in auth_publickey raise SSHException("No existing session") paramiko.ssh_exception.SSHException: No existing session 2024-09-16T12:12:32.388 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-09-16T12:12:33.735 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-09-16T12:12:33.735 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.4.0-195-generic"... 2024-09-16T12:12:33.736 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-09-16T12:12:33.783 INFO:teuthology.orchestra.run.smithi172.stdout:5.4.0-195-generic 2024-09-16T12:12:33.783 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-16T12:12:33.784 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T12:12:33.784 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T12:12:34.785 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-09-16T12:12:34.785 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T12:12:34.862 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2024-09-16T12:12:34.884 DEBUG:teuthology.parallel:result is None 2024-09-16T12:12:38.178 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-09-16T12:12:38.178 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T12:12:38.422 DEBUG:teuthology.orchestra.run.smithi018:> true 2024-09-16T12:12:39.054 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-09-16T12:12:39.054 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.4.0-195-generic"... 2024-09-16T12:12:39.055 DEBUG:teuthology.orchestra.run.smithi018:> uname -r 2024-09-16T12:12:39.061 INFO:teuthology.orchestra.run.smithi018.stdout:5.4.0-195-generic 2024-09-16T12:12:39.061 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-16T12:12:39.061 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T12:12:39.061 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T12:12:40.062 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-09-16T12:12:40.062 DEBUG:teuthology.orchestra.run.smithi018:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T12:12:40.160 INFO:teuthology.orchestra.run.smithi018.stdout:ttyS1 2024-09-16T12:12:40.192 DEBUG:teuthology.parallel:result is None 2024-09-16T12:12:40.193 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T12:12:40.200 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T12:12:40.200 DEBUG:teuthology.orchestra.run.smithi018:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T12:12:40.203 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T12:12:40.210 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T12:12:40.231 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T12:12:40.245 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T12:12:40.245 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T12:12:40.247 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T12:12:40.284 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T12:12:40.290 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T12:12:40.290 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T12:12:40.308 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T12:12:40.323 INFO:teuthology.orchestra.run.smithi018.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T12:12:40.329 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T12:12:40.334 INFO:teuthology.orchestra.run.smithi018.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T12:12:40.340 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T12:12:40.342 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T12:12:40.349 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T12:12:40.349 DEBUG:teuthology.orchestra.run.smithi018:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T12:12:40.351 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T12:12:40.366 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T12:12:40.375 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T12:12:40.375 DEBUG:teuthology.orchestra.run.smithi018:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T12:12:40.412 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T12:12:40.419 DEBUG:teuthology.orchestra.run.smithi018:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T12:12:40.426 DEBUG:teuthology.orchestra.run.smithi018:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T12:12:40.433 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:12:40.433 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T12:12:40.490 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T12:12:40.497 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T12:12:40.544 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:12:40.545 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T12:12:40.569 DEBUG:teuthology.orchestra.run.smithi018:> sudo service rsyslog restart 2024-09-16T12:12:40.571 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2024-09-16T12:12:40.620 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T12:12:40.627 INFO:teuthology.task.internal:Starting timer... 2024-09-16T12:12:40.627 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T12:12:40.711 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T12:12:40.742 DEBUG:teuthology.task.selinux:Excluding smithi018: OS 'ubuntu' does not support SELinux 2024-09-16T12:12:40.742 DEBUG:teuthology.task.selinux:Excluding smithi172: OS 'ubuntu' does not support SELinux 2024-09-16T12:12:40.742 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T12:12:40.742 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-16T12:12:40.742 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T12:12:40.743 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T12:12:40.751 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T12:12:40.762 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T12:12:40.763 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi018.front.sepia.ceph.com,smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T12:18:16.201 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi018.front.sepia.ceph.com'), Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2024-09-16T12:18:16.202 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-09-16T12:18:16.203 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T12:18:16.294 DEBUG:teuthology.orchestra.run.smithi018:> true 2024-09-16T12:18:16.418 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-09-16T12:18:16.419 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-09-16T12:18:16.419 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T12:18:16.513 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-09-16T12:18:16.634 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-09-16T12:18:16.634 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T12:18:16.647 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T12:18:16.648 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T12:18:16.648 DEBUG:teuthology.orchestra.run.smithi018:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T12:18:16.650 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T12:18:16.650 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T12:18:16.681 INFO:teuthology.orchestra.run.smithi018.stdout:16 Sep 12:18:16 ntpd[13664]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-16T12:18:16.681 INFO:teuthology.orchestra.run.smithi018.stdout:16 Sep 12:18:16 ntpd[13664]: Command line: ntpd -gq 2024-09-16T12:18:16.682 INFO:teuthology.orchestra.run.smithi018.stdout:16 Sep 12:18:16 ntpd[13664]: proto: precision = 0.310 usec (-22) 2024-09-16T12:18:16.683 INFO:teuthology.orchestra.run.smithi018.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T12:18:16.683 INFO:teuthology.orchestra.run.smithi018.stdout:16 Sep 12:18:16 ntpd[13664]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T12:18:16.683 INFO:teuthology.orchestra.run.smithi018.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T12:18:16.684 INFO:teuthology.orchestra.run.smithi018.stdout:16 Sep 12:18:16 ntpd[13664]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T12:18:16.684 INFO:teuthology.orchestra.run.smithi018.stdout:16 Sep 12:18:16 ntpd[13664]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T12:18:16.684 INFO:teuthology.orchestra.run.smithi018.stdout:16 Sep 12:18:16 ntpd[13664]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T12:18:16.685 INFO:teuthology.orchestra.run.smithi018.stdout:16 Sep 12:18:16 ntpd[13664]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T12:18:16.685 INFO:teuthology.orchestra.run.smithi018.stdout:16 Sep 12:18:16 ntpd[13664]: Listen normally on 3 ens1f0 172.21.15.18:123 2024-09-16T12:18:16.685 INFO:teuthology.orchestra.run.smithi018.stdout:16 Sep 12:18:16 ntpd[13664]: Listen normally on 4 lo [::1]:123 2024-09-16T12:18:16.685 INFO:teuthology.orchestra.run.smithi018.stdout:16 Sep 12:18:16 ntpd[13664]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1362%3]:123 2024-09-16T12:18:16.685 INFO:teuthology.orchestra.run.smithi018.stdout:16 Sep 12:18:16 ntpd[13664]: Listening on routing socket on fd #22 for interface updates 2024-09-16T12:18:16.685 INFO:teuthology.orchestra.run.smithi172.stdout:16 Sep 12:18:16 ntpd[13742]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-16T12:18:16.685 INFO:teuthology.orchestra.run.smithi172.stdout:16 Sep 12:18:16 ntpd[13742]: Command line: ntpd -gq 2024-09-16T12:18:16.687 INFO:teuthology.orchestra.run.smithi172.stdout:16 Sep 12:18:16 ntpd[13742]: proto: precision = 0.294 usec (-22) 2024-09-16T12:18:16.687 INFO:teuthology.orchestra.run.smithi172.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T12:18:16.687 INFO:teuthology.orchestra.run.smithi172.stdout:16 Sep 12:18:16 ntpd[13742]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T12:18:16.688 INFO:teuthology.orchestra.run.smithi172.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T12:18:16.689 INFO:teuthology.orchestra.run.smithi172.stdout:16 Sep 12:18:16 ntpd[13742]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T12:18:16.689 INFO:teuthology.orchestra.run.smithi172.stdout:16 Sep 12:18:16 ntpd[13742]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T12:18:16.689 INFO:teuthology.orchestra.run.smithi172.stdout:16 Sep 12:18:16 ntpd[13742]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T12:18:16.689 INFO:teuthology.orchestra.run.smithi172.stdout:16 Sep 12:18:16 ntpd[13742]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T12:18:16.689 INFO:teuthology.orchestra.run.smithi172.stdout:16 Sep 12:18:16 ntpd[13742]: Listen normally on 3 enp3s0f1 172.21.15.172:123 2024-09-16T12:18:16.689 INFO:teuthology.orchestra.run.smithi172.stdout:16 Sep 12:18:16 ntpd[13742]: Listen normally on 4 lo [::1]:123 2024-09-16T12:18:16.689 INFO:teuthology.orchestra.run.smithi172.stdout:16 Sep 12:18:16 ntpd[13742]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7f91%5]:123 2024-09-16T12:18:16.689 INFO:teuthology.orchestra.run.smithi172.stdout:16 Sep 12:18:16 ntpd[13742]: Listening on routing socket on fd #22 for interface updates 2024-09-16T12:18:17.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:17 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:17.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:17 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:17.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:17 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:17.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:17 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:17.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:17 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:17.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:17 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:17.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:17 ntpd[13664]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T12:18:17.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:17 ntpd[13664]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T12:18:17.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:17 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:17.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:17 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:17.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:17 ntpd[13664]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T12:18:17.684 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:17 ntpd[13664]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T12:18:17.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:17 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:17.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:17 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:17.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:17 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:17.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:17 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:17.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:17 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:17.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:17 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:17.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:17 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:17.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:17 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:17.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:17 ntpd[13742]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T12:18:17.688 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:17 ntpd[13742]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T12:18:17.688 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:17 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:17.688 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:17 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:17.688 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:17 ntpd[13742]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T12:18:17.688 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:17 ntpd[13742]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T12:18:18.682 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:18 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:18.682 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:18 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:18.682 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:18 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:18.682 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:18 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:18.682 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:18 ntpd[13664]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T12:18:18.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:18 ntpd[13664]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T12:18:18.686 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:18 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:18.686 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:18 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:18.686 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:18 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:18.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:18 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:18.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:18 ntpd[13742]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T12:18:18.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:18 ntpd[13742]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T12:18:19.682 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:19 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:19.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:19 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:19.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:19 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:19.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:19 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:19.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:19 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:19.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:19 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:19.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:19 ntpd[13664]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T12:18:19.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:19 ntpd[13664]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T12:18:19.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:19 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:19.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:19 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:19.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:19 ntpd[13664]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T12:18:19.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:19 ntpd[13664]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T12:18:19.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:19 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:19.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:19 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:19.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:19 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:19.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:19 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:19.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:19 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:19.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:19 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:19.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:19 ntpd[13742]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T12:18:19.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:19 ntpd[13742]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T12:18:19.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:19 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:19.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:19 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:19.688 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:19 ntpd[13742]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T12:18:19.688 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:19 ntpd[13742]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T12:18:20.682 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:20 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:20.682 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:20 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:20.682 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:20 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:20.682 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:20 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:20.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:20 ntpd[13664]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T12:18:20.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:20 ntpd[13664]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T12:18:20.686 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:20 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:20.686 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:20 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:20.686 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:20 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:20.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:20 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:20.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:20 ntpd[13742]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T12:18:20.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:20 ntpd[13742]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T12:18:21.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:21 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:21.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:21 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:21.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:21 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:21.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:21 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:21.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:21 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:21.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:21 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:21.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:21 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:21.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:21 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:21.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:21 ntpd[13664]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T12:18:21.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:21 ntpd[13664]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T12:18:21.684 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:21 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:21.684 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:21 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:21.684 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:21 ntpd[13664]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T12:18:21.684 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:21 ntpd[13664]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T12:18:21.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:21 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:21.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:21 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:21.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:21 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:21.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:21 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:21.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:21 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:21.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:21 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:21.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:21 ntpd[13742]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T12:18:21.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:21 ntpd[13742]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T12:18:21.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:21 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:21.688 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:21 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:21.688 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:21 ntpd[13742]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T12:18:21.688 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:21 ntpd[13742]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T12:18:22.682 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:22 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:22.682 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:22 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:22.682 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:22 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:22.682 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:22 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:22.682 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:22 ntpd[13664]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T12:18:22.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:22 ntpd[13664]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T12:18:22.686 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:22 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:22.686 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:22 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:22.686 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:22 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:22.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:22 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:22.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:22 ntpd[13742]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T12:18:22.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:22 ntpd[13742]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T12:18:23.682 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:23 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:23.682 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:23 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:23.682 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:23 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:23.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:23 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:23.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:23 ntpd[13664]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:23.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:23 ntpd[13664]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:23.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:23 ntpd[13664]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T12:18:23.683 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:23 ntpd[13664]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T12:18:23.683 INFO:teuthology.orchestra.run.smithi018.stdout:16 Sep 12:18:23 ntpd[13664]: ntpd: time slew +0.002006 s 2024-09-16T12:18:23.683 INFO:teuthology.orchestra.run.smithi018.stdout:ntpd: time slew +0.002006s 2024-09-16T12:18:23.684 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:23 ntpd[13664]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T12:18:23.684 INFO:teuthology.orchestra.run.smithi018.stderr:16 Sep 12:18:23 ntpd[13664]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T12:18:23.686 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:23 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:23.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:23 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:23.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:23 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:23.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:23 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:23.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:23 ntpd[13742]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T12:18:23.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:23 ntpd[13742]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T12:18:23.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:23 ntpd[13742]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T12:18:23.687 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:23 ntpd[13742]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T12:18:23.687 INFO:teuthology.orchestra.run.smithi172.stdout:16 Sep 12:18:23 ntpd[13742]: ntpd: time slew +0.002858 s 2024-09-16T12:18:23.687 INFO:teuthology.orchestra.run.smithi172.stdout:ntpd: time slew +0.002858s 2024-09-16T12:18:23.688 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:23 ntpd[13742]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T12:18:23.688 INFO:teuthology.orchestra.run.smithi172.stderr:16 Sep 12:18:23 ntpd[13742]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T12:18:23.747 INFO:teuthology.orchestra.run.smithi018.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T12:18:23.747 INFO:teuthology.orchestra.run.smithi018.stdout:============================================================================== 2024-09-16T12:18:23.747 INFO:teuthology.orchestra.run.smithi018.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T12:18:23.747 INFO:teuthology.orchestra.run.smithi018.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T12:18:23.747 INFO:teuthology.orchestra.run.smithi018.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T12:18:23.747 INFO:teuthology.orchestra.run.smithi018.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T12:18:23.751 INFO:teuthology.orchestra.run.smithi172.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T12:18:23.751 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================== 2024-09-16T12:18:23.751 INFO:teuthology.orchestra.run.smithi172.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T12:18:23.751 INFO:teuthology.orchestra.run.smithi172.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T12:18:23.751 INFO:teuthology.orchestra.run.smithi172.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T12:18:23.751 INFO:teuthology.orchestra.run.smithi172.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-16T12:18:23.752 INFO:teuthology.run_tasks:Running task install... 2024-09-16T12:18:23.760 DEBUG:teuthology.task.install:project ceph 2024-09-16T12:18:23.760 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'extra_system_packages': ['pv']} 2024-09-16T12:18:23.761 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror', 'rbd-nbd'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'extra_system_packages': ['pv']} 2024-09-16T12:18:23.761 INFO:teuthology.task.install:Using flavor: default 2024-09-16T12:18:23.766 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T12:18:23.766 INFO:teuthology.task.install:extra packages: ['rbd-mirror', 'rbd-nbd'] 2024-09-16T12:18:23.767 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt-key list | grep Ceph 2024-09-16T12:18:23.767 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-key list | grep Ceph 2024-09-16T12:18:23.868 INFO:teuthology.orchestra.run.smithi018.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-16T12:18:23.899 INFO:teuthology.orchestra.run.smithi172.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-16T12:18:23.930 INFO:teuthology.orchestra.run.smithi018.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T12:18:23.930 INFO:teuthology.orchestra.run.smithi018.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T12:18:23.931 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror, rbd-nbd on remote deb x86_64 2024-09-16T12:18:23.931 INFO:teuthology.task.install.deb:Installing system (non-project) packages: pv on remote deb x86_64 2024-09-16T12:18:23.931 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T12:18:23.966 INFO:teuthology.orchestra.run.smithi172.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T12:18:23.967 INFO:teuthology.orchestra.run.smithi172.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T12:18:23.967 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror, rbd-nbd on remote deb x86_64 2024-09-16T12:18:23.967 INFO:teuthology.task.install.deb:Installing system (non-project) packages: pv on remote deb x86_64 2024-09-16T12:18:23.967 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T12:18:24.089 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default/ 2024-09-16T12:18:24.089 INFO:teuthology.task.install.deb:Package version is 17.2.7-1643-gd75f2382-1focal 2024-09-16T12:18:24.113 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default/ 2024-09-16T12:18:24.114 INFO:teuthology.task.install.deb:Package version is 17.2.7-1643-gd75f2382-1focal 2024-09-16T12:18:24.206 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:18:24.206 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T12:18:24.222 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:18:24.222 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T12:18:24.223 DEBUG:teuthology.orchestra.run.smithi018:> sudo apt-get update 2024-09-16T12:18:24.239 DEBUG:teuthology.orchestra.run.smithi172:> sudo apt-get update 2024-09-16T12:18:24.431 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T12:18:24.431 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-16T12:18:24.467 INFO:teuthology.orchestra.run.smithi018.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal InRelease 2024-09-16T12:18:24.482 INFO:teuthology.orchestra.run.smithi172.stdout:Ign:2 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal InRelease 2024-09-16T12:18:24.488 INFO:teuthology.orchestra.run.smithi018.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-16T12:18:24.505 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-16T12:18:24.511 INFO:teuthology.orchestra.run.smithi018.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release.gpg 2024-09-16T12:18:24.527 INFO:teuthology.orchestra.run.smithi172.stdout:Ign:4 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal Release.gpg 2024-09-16T12:18:24.530 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T12:18:24.533 INFO:teuthology.orchestra.run.smithi018.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-16T12:18:24.545 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-16T12:18:24.549 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-16T12:18:24.566 INFO:teuthology.orchestra.run.smithi018.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main i386 Packages [3,392 B] 2024-09-16T12:18:24.581 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main i386 Packages [3,392 B] 2024-09-16T12:18:24.611 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T12:18:24.627 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-16T12:18:24.693 INFO:teuthology.orchestra.run.smithi018.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T12:18:24.711 INFO:teuthology.orchestra.run.smithi172.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-16T12:18:25.087 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 28.5 kB in 1s (45.0 kB/s) 2024-09-16T12:18:25.105 INFO:teuthology.orchestra.run.smithi018.stdout:Fetched 28.5 kB in 1s (42.5 kB/s) 2024-09-16T12:18:26.237 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-09-16T12:18:26.262 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1643-gd75f2382-1focal cephadm=17.2.7-1643-gd75f2382-1focal ceph-mds=17.2.7-1643-gd75f2382-1focal ceph-mgr=17.2.7-1643-gd75f2382-1focal ceph-common=17.2.7-1643-gd75f2382-1focal ceph-fuse=17.2.7-1643-gd75f2382-1focal ceph-test=17.2.7-1643-gd75f2382-1focal ceph-volume=17.2.7-1643-gd75f2382-1focal radosgw=17.2.7-1643-gd75f2382-1focal python3-rados=17.2.7-1643-gd75f2382-1focal python3-rgw=17.2.7-1643-gd75f2382-1focal python3-cephfs=17.2.7-1643-gd75f2382-1focal python3-rbd=17.2.7-1643-gd75f2382-1focal libcephfs2=17.2.7-1643-gd75f2382-1focal libcephfs-dev=17.2.7-1643-gd75f2382-1focal librados2=17.2.7-1643-gd75f2382-1focal librbd1=17.2.7-1643-gd75f2382-1focal rbd-fuse=17.2.7-1643-gd75f2382-1focal rbd-mirror=17.2.7-1643-gd75f2382-1focal rbd-nbd=17.2.7-1643-gd75f2382-1focal 2024-09-16T12:18:26.304 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2024-09-16T12:18:26.332 DEBUG:teuthology.orchestra.run.smithi018:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1643-gd75f2382-1focal cephadm=17.2.7-1643-gd75f2382-1focal ceph-mds=17.2.7-1643-gd75f2382-1focal ceph-mgr=17.2.7-1643-gd75f2382-1focal ceph-common=17.2.7-1643-gd75f2382-1focal ceph-fuse=17.2.7-1643-gd75f2382-1focal ceph-test=17.2.7-1643-gd75f2382-1focal ceph-volume=17.2.7-1643-gd75f2382-1focal radosgw=17.2.7-1643-gd75f2382-1focal python3-rados=17.2.7-1643-gd75f2382-1focal python3-rgw=17.2.7-1643-gd75f2382-1focal python3-cephfs=17.2.7-1643-gd75f2382-1focal python3-rbd=17.2.7-1643-gd75f2382-1focal libcephfs2=17.2.7-1643-gd75f2382-1focal libcephfs-dev=17.2.7-1643-gd75f2382-1focal librados2=17.2.7-1643-gd75f2382-1focal librbd1=17.2.7-1643-gd75f2382-1focal rbd-fuse=17.2.7-1643-gd75f2382-1focal rbd-mirror=17.2.7-1643-gd75f2382-1focal rbd-nbd=17.2.7-1643-gd75f2382-1focal 2024-09-16T12:18:26.337 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-09-16T12:18:26.408 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2024-09-16T12:18:26.555 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2024-09-16T12:18:26.555 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2024-09-16T12:18:26.632 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2024-09-16T12:18:26.633 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2024-09-16T12:18:26.752 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T12:18:26.753 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-16T12:18:26.753 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T12:18:26.754 INFO:teuthology.orchestra.run.smithi172.stdout:The following additional packages will be installed: 2024-09-16T12:18:26.754 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T12:18:26.754 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-16T12:18:26.754 INFO:teuthology.orchestra.run.smithi172.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-16T12:18:26.754 INFO:teuthology.orchestra.run.smithi172.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-16T12:18:26.754 INFO:teuthology.orchestra.run.smithi172.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-16T12:18:26.755 INFO:teuthology.orchestra.run.smithi172.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-16T12:18:26.755 INFO:teuthology.orchestra.run.smithi172.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-16T12:18:26.755 INFO:teuthology.orchestra.run.smithi172.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-16T12:18:26.755 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-16T12:18:26.755 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-16T12:18:26.755 INFO:teuthology.orchestra.run.smithi172.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-16T12:18:26.755 INFO:teuthology.orchestra.run.smithi172.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-16T12:18:26.755 INFO:teuthology.orchestra.run.smithi172.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-16T12:18:26.755 INFO:teuthology.orchestra.run.smithi172.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-16T12:18:26.755 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-16T12:18:26.755 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T12:18:26.755 INFO:teuthology.orchestra.run.smithi172.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-16T12:18:26.755 INFO:teuthology.orchestra.run.smithi172.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T12:18:26.756 INFO:teuthology.orchestra.run.smithi172.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-16T12:18:26.756 INFO:teuthology.orchestra.run.smithi172.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-16T12:18:26.756 INFO:teuthology.orchestra.run.smithi172.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-16T12:18:26.756 INFO:teuthology.orchestra.run.smithi172.stdout: socat xmlstarlet 2024-09-16T12:18:26.756 INFO:teuthology.orchestra.run.smithi172.stdout:Suggested packages: 2024-09-16T12:18:26.756 INFO:teuthology.orchestra.run.smithi172.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-16T12:18:26.756 INFO:teuthology.orchestra.run.smithi172.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-16T12:18:26.757 INFO:teuthology.orchestra.run.smithi172.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-16T12:18:26.757 INFO:teuthology.orchestra.run.smithi172.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-16T12:18:26.757 INFO:teuthology.orchestra.run.smithi172.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-16T12:18:26.757 INFO:teuthology.orchestra.run.smithi172.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-16T12:18:26.757 INFO:teuthology.orchestra.run.smithi172.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-16T12:18:26.757 INFO:teuthology.orchestra.run.smithi172.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T12:18:26.757 INFO:teuthology.orchestra.run.smithi172.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-16T12:18:26.757 INFO:teuthology.orchestra.run.smithi172.stdout:Recommended packages: 2024-09-16T12:18:26.757 INFO:teuthology.orchestra.run.smithi172.stdout: btrfs-tools 2024-09-16T12:18:26.840 INFO:teuthology.orchestra.run.smithi018.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T12:18:26.841 INFO:teuthology.orchestra.run.smithi018.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-16T12:18:26.841 INFO:teuthology.orchestra.run.smithi018.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T12:18:26.841 INFO:teuthology.orchestra.run.smithi018.stdout:The following additional packages will be installed: 2024-09-16T12:18:26.842 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T12:18:26.842 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-16T12:18:26.842 INFO:teuthology.orchestra.run.smithi018.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-16T12:18:26.842 INFO:teuthology.orchestra.run.smithi018.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-16T12:18:26.842 INFO:teuthology.orchestra.run.smithi018.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-16T12:18:26.843 INFO:teuthology.orchestra.run.smithi018.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-16T12:18:26.843 INFO:teuthology.orchestra.run.smithi018.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-16T12:18:26.843 INFO:teuthology.orchestra.run.smithi018.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-16T12:18:26.843 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-16T12:18:26.843 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-16T12:18:26.843 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-16T12:18:26.843 INFO:teuthology.orchestra.run.smithi018.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-16T12:18:26.844 INFO:teuthology.orchestra.run.smithi018.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-16T12:18:26.844 INFO:teuthology.orchestra.run.smithi018.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-16T12:18:26.844 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-16T12:18:26.844 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T12:18:26.844 INFO:teuthology.orchestra.run.smithi018.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-16T12:18:26.844 INFO:teuthology.orchestra.run.smithi018.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T12:18:26.844 INFO:teuthology.orchestra.run.smithi018.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-16T12:18:26.844 INFO:teuthology.orchestra.run.smithi018.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-16T12:18:26.844 INFO:teuthology.orchestra.run.smithi018.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-16T12:18:26.844 INFO:teuthology.orchestra.run.smithi018.stdout: socat xmlstarlet 2024-09-16T12:18:26.845 INFO:teuthology.orchestra.run.smithi018.stdout:Suggested packages: 2024-09-16T12:18:26.845 INFO:teuthology.orchestra.run.smithi018.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-16T12:18:26.845 INFO:teuthology.orchestra.run.smithi018.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-16T12:18:26.845 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-16T12:18:26.845 INFO:teuthology.orchestra.run.smithi018.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-16T12:18:26.845 INFO:teuthology.orchestra.run.smithi018.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-16T12:18:26.845 INFO:teuthology.orchestra.run.smithi018.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-16T12:18:26.845 INFO:teuthology.orchestra.run.smithi018.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-16T12:18:26.845 INFO:teuthology.orchestra.run.smithi018.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-16T12:18:26.845 INFO:teuthology.orchestra.run.smithi018.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-16T12:18:26.846 INFO:teuthology.orchestra.run.smithi018.stdout:Recommended packages: 2024-09-16T12:18:26.846 INFO:teuthology.orchestra.run.smithi018.stdout: btrfs-tools 2024-09-16T12:18:26.883 INFO:teuthology.orchestra.run.smithi172.stdout:The following NEW packages will be installed: 2024-09-16T12:18:26.883 INFO:teuthology.orchestra.run.smithi172.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T12:18:26.883 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T12:18:26.883 INFO:teuthology.orchestra.run.smithi172.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-16T12:18:26.884 INFO:teuthology.orchestra.run.smithi172.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-16T12:18:26.884 INFO:teuthology.orchestra.run.smithi172.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-16T12:18:26.884 INFO:teuthology.orchestra.run.smithi172.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T12:18:26.884 INFO:teuthology.orchestra.run.smithi172.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-16T12:18:26.884 INFO:teuthology.orchestra.run.smithi172.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-16T12:18:26.885 INFO:teuthology.orchestra.run.smithi172.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-16T12:18:26.885 INFO:teuthology.orchestra.run.smithi172.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-16T12:18:26.885 INFO:teuthology.orchestra.run.smithi172.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-16T12:18:26.885 INFO:teuthology.orchestra.run.smithi172.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-16T12:18:26.885 INFO:teuthology.orchestra.run.smithi172.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-16T12:18:26.885 INFO:teuthology.orchestra.run.smithi172.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-16T12:18:26.885 INFO:teuthology.orchestra.run.smithi172.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-16T12:18:26.885 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-16T12:18:26.885 INFO:teuthology.orchestra.run.smithi172.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T12:18:26.885 INFO:teuthology.orchestra.run.smithi172.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T12:18:26.885 INFO:teuthology.orchestra.run.smithi172.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-16T12:18:26.885 INFO:teuthology.orchestra.run.smithi172.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-16T12:18:26.886 INFO:teuthology.orchestra.run.smithi172.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-16T12:18:26.886 INFO:teuthology.orchestra.run.smithi172.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-16T12:18:26.886 INFO:teuthology.orchestra.run.smithi172.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-16T12:18:26.886 INFO:teuthology.orchestra.run.smithi172.stdout: qttranslations5-l10n radosgw rbd-fuse rbd-mirror rbd-nbd socat xmlstarlet 2024-09-16T12:18:26.886 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages will be upgraded: 2024-09-16T12:18:26.886 INFO:teuthology.orchestra.run.smithi172.stdout: librados2 librbd1 2024-09-16T12:18:26.952 INFO:teuthology.orchestra.run.smithi172.stdout:2 upgraded, 110 newly installed, 0 to remove and 295 not upgraded. 2024-09-16T12:18:26.953 INFO:teuthology.orchestra.run.smithi172.stdout:Need to get 205 MB of archives. 2024-09-16T12:18:26.953 INFO:teuthology.orchestra.run.smithi172.stdout:After this operation, 955 MB of additional disk space will be used. 2024-09-16T12:18:26.953 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-16T12:18:26.972 INFO:teuthology.orchestra.run.smithi018.stdout:The following NEW packages will be installed: 2024-09-16T12:18:26.972 INFO:teuthology.orchestra.run.smithi018.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T12:18:26.972 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T12:18:26.972 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-16T12:18:26.972 INFO:teuthology.orchestra.run.smithi018.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-16T12:18:26.972 INFO:teuthology.orchestra.run.smithi018.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-16T12:18:26.972 INFO:teuthology.orchestra.run.smithi018.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-16T12:18:26.973 INFO:teuthology.orchestra.run.smithi018.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-16T12:18:26.973 INFO:teuthology.orchestra.run.smithi018.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-16T12:18:26.973 INFO:teuthology.orchestra.run.smithi018.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-16T12:18:26.973 INFO:teuthology.orchestra.run.smithi018.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-16T12:18:26.974 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-16T12:18:26.974 INFO:teuthology.orchestra.run.smithi018.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-16T12:18:26.974 INFO:teuthology.orchestra.run.smithi018.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-16T12:18:26.974 INFO:teuthology.orchestra.run.smithi018.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-16T12:18:26.974 INFO:teuthology.orchestra.run.smithi018.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-16T12:18:26.974 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-16T12:18:26.974 INFO:teuthology.orchestra.run.smithi018.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T12:18:26.974 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-16T12:18:26.974 INFO:teuthology.orchestra.run.smithi018.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-16T12:18:26.974 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-16T12:18:26.974 INFO:teuthology.orchestra.run.smithi018.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-16T12:18:26.974 INFO:teuthology.orchestra.run.smithi018.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-16T12:18:26.974 INFO:teuthology.orchestra.run.smithi018.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-16T12:18:26.974 INFO:teuthology.orchestra.run.smithi018.stdout: qttranslations5-l10n radosgw rbd-fuse rbd-mirror rbd-nbd socat xmlstarlet 2024-09-16T12:18:26.975 INFO:teuthology.orchestra.run.smithi018.stdout:The following packages will be upgraded: 2024-09-16T12:18:26.975 INFO:teuthology.orchestra.run.smithi018.stdout: librados2 librbd1 2024-09-16T12:18:27.036 INFO:teuthology.orchestra.run.smithi172.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1643-gd75f2382-1focal [3,508 kB] 2024-09-16T12:18:27.036 INFO:teuthology.orchestra.run.smithi172.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-16T12:18:27.039 INFO:teuthology.orchestra.run.smithi172.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-16T12:18:27.047 INFO:teuthology.orchestra.run.smithi172.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-16T12:18:27.056 INFO:teuthology.orchestra.run.smithi172.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-16T12:18:27.079 INFO:teuthology.orchestra.run.smithi172.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-16T12:18:27.081 INFO:teuthology.orchestra.run.smithi172.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-16T12:18:27.088 INFO:teuthology.orchestra.run.smithi172.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-16T12:18:27.121 INFO:teuthology.orchestra.run.smithi172.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-16T12:18:27.123 INFO:teuthology.orchestra.run.smithi172.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-16T12:18:27.126 INFO:teuthology.orchestra.run.smithi018.stdout:2 upgraded, 110 newly installed, 0 to remove and 295 not upgraded. 2024-09-16T12:18:27.126 INFO:teuthology.orchestra.run.smithi018.stdout:Need to get 205 MB of archives. 2024-09-16T12:18:27.126 INFO:teuthology.orchestra.run.smithi018.stdout:After this operation, 955 MB of additional disk space will be used. 2024-09-16T12:18:27.126 INFO:teuthology.orchestra.run.smithi018.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1643-gd75f2382-1focal [3,508 kB] 2024-09-16T12:18:27.129 INFO:teuthology.orchestra.run.smithi172.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-16T12:18:27.131 INFO:teuthology.orchestra.run.smithi172.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-16T12:18:27.131 INFO:teuthology.orchestra.run.smithi172.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-16T12:18:27.133 INFO:teuthology.orchestra.run.smithi172.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-16T12:18:27.134 INFO:teuthology.orchestra.run.smithi172.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-16T12:18:27.134 INFO:teuthology.orchestra.run.smithi172.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-16T12:18:27.134 INFO:teuthology.orchestra.run.smithi172.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-16T12:18:27.139 INFO:teuthology.orchestra.run.smithi172.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-16T12:18:27.153 INFO:teuthology.orchestra.run.smithi172.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-16T12:18:27.154 INFO:teuthology.orchestra.run.smithi172.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-16T12:18:27.154 INFO:teuthology.orchestra.run.smithi172.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-16T12:18:27.155 INFO:teuthology.orchestra.run.smithi172.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-16T12:18:27.156 INFO:teuthology.orchestra.run.smithi172.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T12:18:27.156 INFO:teuthology.orchestra.run.smithi172.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-16T12:18:27.157 INFO:teuthology.orchestra.run.smithi172.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-16T12:18:27.158 INFO:teuthology.orchestra.run.smithi172.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-16T12:18:27.158 INFO:teuthology.orchestra.run.smithi172.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-16T12:18:27.168 INFO:teuthology.orchestra.run.smithi018.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-16T12:18:27.173 INFO:teuthology.orchestra.run.smithi172.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-16T12:18:27.175 INFO:teuthology.orchestra.run.smithi172.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-16T12:18:27.175 INFO:teuthology.orchestra.run.smithi172.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-16T12:18:27.190 INFO:teuthology.orchestra.run.smithi172.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-16T12:18:27.190 INFO:teuthology.orchestra.run.smithi172.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-16T12:18:27.190 INFO:teuthology.orchestra.run.smithi172.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-16T12:18:27.191 INFO:teuthology.orchestra.run.smithi172.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-16T12:18:27.192 INFO:teuthology.orchestra.run.smithi172.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-16T12:18:27.194 INFO:teuthology.orchestra.run.smithi172.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-16T12:18:27.195 INFO:teuthology.orchestra.run.smithi172.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T12:18:27.207 INFO:teuthology.orchestra.run.smithi172.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-16T12:18:27.209 INFO:teuthology.orchestra.run.smithi172.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-16T12:18:27.211 INFO:teuthology.orchestra.run.smithi172.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-16T12:18:27.211 INFO:teuthology.orchestra.run.smithi172.stdout:Get:42 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1643-gd75f2382-1focal [3,564 kB] 2024-09-16T12:18:27.224 INFO:teuthology.orchestra.run.smithi172.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-16T12:18:27.225 INFO:teuthology.orchestra.run.smithi172.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-16T12:18:27.225 INFO:teuthology.orchestra.run.smithi172.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-16T12:18:27.292 INFO:teuthology.orchestra.run.smithi172.stdout:Get:46 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1643-gd75f2382-1focal [761 kB] 2024-09-16T12:18:27.312 INFO:teuthology.orchestra.run.smithi172.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-16T12:18:27.322 INFO:teuthology.orchestra.run.smithi172.stdout:Get:48 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1643-gd75f2382-1focal [414 kB] 2024-09-16T12:18:27.328 INFO:teuthology.orchestra.run.smithi172.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-16T12:18:27.355 INFO:teuthology.orchestra.run.smithi018.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1643-gd75f2382-1focal [3,564 kB] 2024-09-16T12:18:27.407 INFO:teuthology.orchestra.run.smithi172.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-16T12:18:27.407 INFO:teuthology.orchestra.run.smithi172.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1643-gd75f2382-1focal [22.6 kB] 2024-09-16T12:18:27.407 INFO:teuthology.orchestra.run.smithi172.stdout:Get:52 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1643-gd75f2382-1focal [200 kB] 2024-09-16T12:18:27.410 INFO:teuthology.orchestra.run.smithi172.stdout:Get:53 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1643-gd75f2382-1focal [49.1 kB] 2024-09-16T12:18:27.411 INFO:teuthology.orchestra.run.smithi172.stdout:Get:54 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1643-gd75f2382-1focal [389 kB] 2024-09-16T12:18:27.418 INFO:teuthology.orchestra.run.smithi172.stdout:Get:55 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1643-gd75f2382-1focal [5,637 kB] 2024-09-16T12:18:27.426 INFO:teuthology.orchestra.run.smithi172.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-16T12:18:27.426 INFO:teuthology.orchestra.run.smithi172.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-16T12:18:27.427 INFO:teuthology.orchestra.run.smithi172.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-16T12:18:27.427 INFO:teuthology.orchestra.run.smithi172.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-16T12:18:27.428 INFO:teuthology.orchestra.run.smithi172.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-16T12:18:27.428 INFO:teuthology.orchestra.run.smithi172.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-16T12:18:27.428 INFO:teuthology.orchestra.run.smithi172.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-16T12:18:27.429 INFO:teuthology.orchestra.run.smithi172.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-16T12:18:27.430 INFO:teuthology.orchestra.run.smithi172.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-16T12:18:27.444 INFO:teuthology.orchestra.run.smithi172.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-16T12:18:27.446 INFO:teuthology.orchestra.run.smithi172.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-16T12:18:27.460 INFO:teuthology.orchestra.run.smithi018.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1643-gd75f2382-1focal [761 kB] 2024-09-16T12:18:27.461 INFO:teuthology.orchestra.run.smithi172.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-16T12:18:27.465 INFO:teuthology.orchestra.run.smithi172.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-16T12:18:27.467 INFO:teuthology.orchestra.run.smithi172.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-16T12:18:27.471 INFO:teuthology.orchestra.run.smithi172.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-16T12:18:27.472 INFO:teuthology.orchestra.run.smithi018.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1643-gd75f2382-1focal [414 kB] 2024-09-16T12:18:27.473 INFO:teuthology.orchestra.run.smithi172.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-16T12:18:27.481 INFO:teuthology.orchestra.run.smithi018.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1643-gd75f2382-1focal [22.6 kB] 2024-09-16T12:18:27.481 INFO:teuthology.orchestra.run.smithi018.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1643-gd75f2382-1focal [200 kB] 2024-09-16T12:18:27.488 INFO:teuthology.orchestra.run.smithi172.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-16T12:18:27.491 INFO:teuthology.orchestra.run.smithi172.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-16T12:18:27.509 INFO:teuthology.orchestra.run.smithi018.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1643-gd75f2382-1focal [49.1 kB] 2024-09-16T12:18:27.510 INFO:teuthology.orchestra.run.smithi018.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1643-gd75f2382-1focal [389 kB] 2024-09-16T12:18:27.517 INFO:teuthology.orchestra.run.smithi018.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1643-gd75f2382-1focal [5,637 kB] 2024-09-16T12:18:27.533 INFO:teuthology.orchestra.run.smithi172.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-16T12:18:27.534 INFO:teuthology.orchestra.run.smithi172.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-16T12:18:27.534 INFO:teuthology.orchestra.run.smithi172.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-16T12:18:27.534 INFO:teuthology.orchestra.run.smithi172.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-16T12:18:27.535 INFO:teuthology.orchestra.run.smithi172.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-16T12:18:27.536 INFO:teuthology.orchestra.run.smithi172.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-16T12:18:27.567 INFO:teuthology.orchestra.run.smithi018.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-16T12:18:27.572 INFO:teuthology.orchestra.run.smithi172.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T12:18:27.572 INFO:teuthology.orchestra.run.smithi172.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1643-gd75f2382-1focal [116 kB] 2024-09-16T12:18:27.573 INFO:teuthology.orchestra.run.smithi172.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-16T12:18:27.573 INFO:teuthology.orchestra.run.smithi172.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-16T12:18:27.574 INFO:teuthology.orchestra.run.smithi018.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-16T12:18:27.574 INFO:teuthology.orchestra.run.smithi172.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1643-gd75f2382-1focal [479 kB] 2024-09-16T12:18:27.575 INFO:teuthology.orchestra.run.smithi172.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-16T12:18:27.576 INFO:teuthology.orchestra.run.smithi172.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-16T12:18:27.610 INFO:teuthology.orchestra.run.smithi172.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-16T12:18:27.610 INFO:teuthology.orchestra.run.smithi172.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1643-gd75f2382-1focal [27.0 MB] 2024-09-16T12:18:27.611 INFO:teuthology.orchestra.run.smithi172.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-16T12:18:27.613 INFO:teuthology.orchestra.run.smithi172.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-16T12:18:27.614 INFO:teuthology.orchestra.run.smithi172.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-16T12:18:27.617 INFO:teuthology.orchestra.run.smithi018.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-16T12:18:27.631 INFO:teuthology.orchestra.run.smithi018.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-16T12:18:27.699 INFO:teuthology.orchestra.run.smithi018.stdout:Get:15 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1643-gd75f2382-1focal [116 kB] 2024-09-16T12:18:27.700 INFO:teuthology.orchestra.run.smithi018.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1643-gd75f2382-1focal [479 kB] 2024-09-16T12:18:27.708 INFO:teuthology.orchestra.run.smithi018.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1643-gd75f2382-1focal [27.0 MB] 2024-09-16T12:18:27.756 INFO:teuthology.orchestra.run.smithi018.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-16T12:18:27.762 INFO:teuthology.orchestra.run.smithi018.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-16T12:18:27.791 INFO:teuthology.orchestra.run.smithi018.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-16T12:18:27.932 INFO:teuthology.orchestra.run.smithi018.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-16T12:18:27.940 INFO:teuthology.orchestra.run.smithi018.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-16T12:18:27.962 INFO:teuthology.orchestra.run.smithi018.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-16T12:18:27.969 INFO:teuthology.orchestra.run.smithi018.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-16T12:18:27.970 INFO:teuthology.orchestra.run.smithi018.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-16T12:18:27.977 INFO:teuthology.orchestra.run.smithi018.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-16T12:18:27.978 INFO:teuthology.orchestra.run.smithi018.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-16T12:18:27.979 INFO:teuthology.orchestra.run.smithi018.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-16T12:18:27.979 INFO:teuthology.orchestra.run.smithi018.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-16T12:18:28.014 INFO:teuthology.orchestra.run.smithi018.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-16T12:18:28.066 INFO:teuthology.orchestra.run.smithi018.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-16T12:18:28.067 INFO:teuthology.orchestra.run.smithi018.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-16T12:18:28.067 INFO:teuthology.orchestra.run.smithi018.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-16T12:18:28.068 INFO:teuthology.orchestra.run.smithi018.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-16T12:18:28.070 INFO:teuthology.orchestra.run.smithi018.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T12:18:28.070 INFO:teuthology.orchestra.run.smithi018.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-16T12:18:28.071 INFO:teuthology.orchestra.run.smithi018.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-16T12:18:28.073 INFO:teuthology.orchestra.run.smithi018.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-16T12:18:28.096 INFO:teuthology.orchestra.run.smithi018.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-16T12:18:28.177 INFO:teuthology.orchestra.run.smithi018.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-16T12:18:28.179 INFO:teuthology.orchestra.run.smithi018.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-16T12:18:28.180 INFO:teuthology.orchestra.run.smithi018.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-16T12:18:28.187 INFO:teuthology.orchestra.run.smithi018.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-16T12:18:28.188 INFO:teuthology.orchestra.run.smithi018.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-16T12:18:28.189 INFO:teuthology.orchestra.run.smithi018.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-16T12:18:28.189 INFO:teuthology.orchestra.run.smithi018.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-16T12:18:28.190 INFO:teuthology.orchestra.run.smithi018.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-16T12:18:28.258 INFO:teuthology.orchestra.run.smithi018.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-16T12:18:28.261 INFO:teuthology.orchestra.run.smithi018.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T12:18:28.340 INFO:teuthology.orchestra.run.smithi018.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-16T12:18:28.342 INFO:teuthology.orchestra.run.smithi018.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-16T12:18:28.344 INFO:teuthology.orchestra.run.smithi018.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-16T12:18:28.346 INFO:teuthology.orchestra.run.smithi018.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-16T12:18:28.346 INFO:teuthology.orchestra.run.smithi018.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-16T12:18:28.347 INFO:teuthology.orchestra.run.smithi018.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-16T12:18:28.553 INFO:teuthology.orchestra.run.smithi172.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1643-gd75f2382-1focal [5,178 kB] 2024-09-16T12:18:28.612 INFO:teuthology.orchestra.run.smithi172.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1643-gd75f2382-1focal [223 kB] 2024-09-16T12:18:28.614 INFO:teuthology.orchestra.run.smithi172.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1643-gd75f2382-1focal [157 kB] 2024-09-16T12:18:28.619 INFO:teuthology.orchestra.run.smithi172.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1643-gd75f2382-1focal [1,594 kB] 2024-09-16T12:18:28.659 INFO:teuthology.orchestra.run.smithi172.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1643-gd75f2382-1focal [4,684 kB] 2024-09-16T12:18:28.763 INFO:teuthology.orchestra.run.smithi018.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-16T12:18:28.781 INFO:teuthology.orchestra.run.smithi172.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1643-gd75f2382-1focal [23.1 MB] 2024-09-16T12:18:28.811 INFO:teuthology.orchestra.run.smithi018.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-16T12:18:28.811 INFO:teuthology.orchestra.run.smithi018.stdout:Get:58 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1643-gd75f2382-1focal [5,178 kB] 2024-09-16T12:18:28.812 INFO:teuthology.orchestra.run.smithi018.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-16T12:18:28.868 INFO:teuthology.orchestra.run.smithi018.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-16T12:18:28.868 INFO:teuthology.orchestra.run.smithi018.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-16T12:18:28.869 INFO:teuthology.orchestra.run.smithi018.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-16T12:18:28.869 INFO:teuthology.orchestra.run.smithi018.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-16T12:18:28.870 INFO:teuthology.orchestra.run.smithi018.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-16T12:18:28.871 INFO:teuthology.orchestra.run.smithi018.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-16T12:18:28.872 INFO:teuthology.orchestra.run.smithi018.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-16T12:18:28.875 INFO:teuthology.orchestra.run.smithi018.stdout:Get:67 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1643-gd75f2382-1focal [223 kB] 2024-09-16T12:18:28.883 INFO:teuthology.orchestra.run.smithi018.stdout:Get:68 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1643-gd75f2382-1focal [157 kB] 2024-09-16T12:18:28.894 INFO:teuthology.orchestra.run.smithi018.stdout:Get:69 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1643-gd75f2382-1focal [1,594 kB] 2024-09-16T12:18:28.927 INFO:teuthology.orchestra.run.smithi018.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-16T12:18:28.928 INFO:teuthology.orchestra.run.smithi018.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-16T12:18:28.946 INFO:teuthology.orchestra.run.smithi018.stdout:Get:72 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1643-gd75f2382-1focal [4,684 kB] 2024-09-16T12:18:29.008 INFO:teuthology.orchestra.run.smithi018.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-16T12:18:29.013 INFO:teuthology.orchestra.run.smithi018.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-16T12:18:29.017 INFO:teuthology.orchestra.run.smithi018.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-16T12:18:29.019 INFO:teuthology.orchestra.run.smithi018.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-16T12:18:29.021 INFO:teuthology.orchestra.run.smithi018.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-16T12:18:29.023 INFO:teuthology.orchestra.run.smithi018.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-16T12:18:29.025 INFO:teuthology.orchestra.run.smithi018.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-16T12:18:29.088 INFO:teuthology.orchestra.run.smithi018.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1643-gd75f2382-1focal [23.1 MB] 2024-09-16T12:18:29.098 INFO:teuthology.orchestra.run.smithi018.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-16T12:18:29.103 INFO:teuthology.orchestra.run.smithi018.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-16T12:18:29.264 INFO:teuthology.orchestra.run.smithi018.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-16T12:18:29.265 INFO:teuthology.orchestra.run.smithi018.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-16T12:18:29.266 INFO:teuthology.orchestra.run.smithi018.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-16T12:18:29.266 INFO:teuthology.orchestra.run.smithi018.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-16T12:18:29.266 INFO:teuthology.orchestra.run.smithi018.stdout:Get:87 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-16T12:18:29.267 INFO:teuthology.orchestra.run.smithi018.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-16T12:18:29.281 INFO:teuthology.orchestra.run.smithi018.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T12:18:29.281 INFO:teuthology.orchestra.run.smithi018.stdout:Get:90 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-16T12:18:29.282 INFO:teuthology.orchestra.run.smithi018.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-16T12:18:29.346 INFO:teuthology.orchestra.run.smithi018.stdout:Get:92 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-16T12:18:29.428 INFO:teuthology.orchestra.run.smithi018.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-16T12:18:29.428 INFO:teuthology.orchestra.run.smithi018.stdout:Get:94 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-16T12:18:29.429 INFO:teuthology.orchestra.run.smithi018.stdout:Get:95 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-16T12:18:29.432 INFO:teuthology.orchestra.run.smithi018.stdout:Get:96 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-16T12:18:29.433 INFO:teuthology.orchestra.run.smithi018.stdout:Get:97 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-16T12:18:29.646 INFO:teuthology.orchestra.run.smithi172.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1643-gd75f2382-1focal [3,940 B] 2024-09-16T12:18:29.647 INFO:teuthology.orchestra.run.smithi172.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1643-gd75f2382-1focal [871 kB] 2024-09-16T12:18:29.661 INFO:teuthology.orchestra.run.smithi172.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1643-gd75f2382-1focal [2,418 kB] 2024-09-16T12:18:29.696 INFO:teuthology.orchestra.run.smithi172.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1643-gd75f2382-1focal [81.4 kB] 2024-09-16T12:18:29.696 INFO:teuthology.orchestra.run.smithi172.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1643-gd75f2382-1focal [120 kB] 2024-09-16T12:18:29.719 INFO:teuthology.orchestra.run.smithi172.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1643-gd75f2382-1focal [6,653 kB] 2024-09-16T12:18:29.825 INFO:teuthology.orchestra.run.smithi018.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1643-gd75f2382-1focal [3,940 B] 2024-09-16T12:18:29.825 INFO:teuthology.orchestra.run.smithi018.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1643-gd75f2382-1focal [871 kB] 2024-09-16T12:18:29.839 INFO:teuthology.orchestra.run.smithi018.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1643-gd75f2382-1focal [2,418 kB] 2024-09-16T12:18:29.856 INFO:teuthology.orchestra.run.smithi018.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1643-gd75f2382-1focal [81.4 kB] 2024-09-16T12:18:29.857 INFO:teuthology.orchestra.run.smithi018.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1643-gd75f2382-1focal [120 kB] 2024-09-16T12:18:29.858 INFO:teuthology.orchestra.run.smithi018.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1643-gd75f2382-1focal [6,653 kB] 2024-09-16T12:18:29.988 INFO:teuthology.orchestra.run.smithi172.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1643-gd75f2382-1focal [7,885 kB] 2024-09-16T12:18:30.040 INFO:teuthology.orchestra.run.smithi018.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1643-gd75f2382-1focal [7,885 kB] 2024-09-16T12:18:30.256 INFO:teuthology.orchestra.run.smithi018.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1643-gd75f2382-1focal [4,064 B] 2024-09-16T12:18:30.256 INFO:teuthology.orchestra.run.smithi018.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1643-gd75f2382-1focal [53.4 MB] 2024-09-16T12:18:30.265 INFO:teuthology.orchestra.run.smithi172.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1643-gd75f2382-1focal [4,064 B] 2024-09-16T12:18:30.265 INFO:teuthology.orchestra.run.smithi172.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1643-gd75f2382-1focal [53.4 MB] 2024-09-16T12:18:31.935 INFO:teuthology.orchestra.run.smithi018.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1643-gd75f2382-1focal [231 kB] 2024-09-16T12:18:31.941 INFO:teuthology.orchestra.run.smithi018.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1643-gd75f2382-1focal [21.9 kB] 2024-09-16T12:18:31.941 INFO:teuthology.orchestra.run.smithi018.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1643-gd75f2382-1focal [17.2 MB] 2024-09-16T12:18:32.372 INFO:teuthology.orchestra.run.smithi018.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1643-gd75f2382-1focal [80.7 kB] 2024-09-16T12:18:32.374 INFO:teuthology.orchestra.run.smithi018.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 rbd-mirror amd64 17.2.7-1643-gd75f2382-1focal [3,537 kB] 2024-09-16T12:18:32.405 INFO:teuthology.orchestra.run.smithi018.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 rbd-nbd amd64 17.2.7-1643-gd75f2382-1focal [166 kB] 2024-09-16T12:18:32.544 INFO:teuthology.orchestra.run.smithi172.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1643-gd75f2382-1focal [231 kB] 2024-09-16T12:18:32.548 INFO:teuthology.orchestra.run.smithi172.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1643-gd75f2382-1focal [21.9 kB] 2024-09-16T12:18:32.549 INFO:teuthology.orchestra.run.smithi172.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1643-gd75f2382-1focal [17.2 MB] 2024-09-16T12:18:32.890 INFO:teuthology.orchestra.run.smithi172.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1643-gd75f2382-1focal [80.7 kB] 2024-09-16T12:18:32.891 INFO:teuthology.orchestra.run.smithi172.stdout:Get:111 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 rbd-mirror amd64 17.2.7-1643-gd75f2382-1focal [3,537 kB] 2024-09-16T12:18:32.933 INFO:teuthology.orchestra.run.smithi172.stdout:Get:112 https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/ubuntu/focal/flavors/default focal/main amd64 rbd-nbd amd64 17.2.7-1643-gd75f2382-1focal [166 kB] 2024-09-16T12:18:33.073 INFO:teuthology.orchestra.run.smithi018.stdout:Fetched 205 MB in 5s (37.8 MB/s) 2024-09-16T12:18:33.216 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-16T12:18:33.273 INFO:teuthology.orchestra.run.smithi018.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-16T12:18:33.278 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-16T12:18:33.356 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T12:18:33.581 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 205 MB in 6s (33.9 MB/s) 2024-09-16T12:18:33.718 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../001-librbd1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:33.723 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-16T12:18:33.778 INFO:teuthology.orchestra.run.smithi172.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-16T12:18:33.782 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-16T12:18:33.844 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking librbd1 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T12:18:33.855 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T12:18:34.192 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../001-librbd1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:34.317 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librbd1 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T12:18:34.508 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../002-librados2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:34.615 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking librados2 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T12:18:35.063 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../002-librados2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:35.197 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librados2 (17.2.7-1643-gd75f2382-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-16T12:18:35.294 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T12:18:35.314 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:35.352 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:35.688 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-rados. 2024-09-16T12:18:35.704 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:35.746 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:35.834 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T12:18:35.851 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:35.883 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:36.057 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T12:18:36.075 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T12:18:36.107 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:36.261 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rados. 2024-09-16T12:18:36.277 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:36.319 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:36.334 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T12:18:36.351 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:36.393 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:36.714 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T12:18:36.729 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T12:18:36.729 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T12:18:36.745 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T12:18:36.772 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:36.787 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:37.049 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T12:18:37.066 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:37.106 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T12:18:37.107 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:37.124 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-16T12:18:37.165 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-16T12:18:37.418 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T12:18:37.418 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T12:18:37.437 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:37.438 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T12:18:37.476 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:37.476 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:37.829 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-16T12:18:37.838 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T12:18:37.845 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-16T12:18:37.858 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-16T12:18:37.887 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T12:18:37.896 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-16T12:18:38.165 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T12:18:38.181 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:38.223 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:38.231 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-16T12:18:38.248 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-16T12:18:38.290 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T12:18:38.588 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-16T12:18:38.607 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-16T12:18:38.618 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T12:18:38.635 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-16T12:18:38.651 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T12:18:38.676 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T12:18:39.004 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-16T12:18:39.021 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-16T12:18:39.054 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T12:18:39.062 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T12:18:39.070 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-16T12:18:39.112 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T12:18:39.390 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T12:18:39.408 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-16T12:18:39.415 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T12:18:39.432 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-16T12:18:39.440 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T12:18:39.473 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T12:18:39.893 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T12:18:39.902 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T12:18:39.910 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-16T12:18:39.918 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T12:18:39.951 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T12:18:39.971 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T12:18:40.279 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T12:18:40.299 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-16T12:18:40.328 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T12:18:40.497 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T12:18:40.513 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T12:18:40.555 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T12:18:40.849 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T12:18:40.865 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T12:18:40.891 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T12:18:40.910 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T12:18:40.917 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T12:18:40.949 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T12:18:41.353 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-16T12:18:41.369 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-16T12:18:41.411 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T12:18:41.493 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T12:18:41.509 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T12:18:41.551 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T12:18:41.764 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package librgw2. 2024-09-16T12:18:41.780 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../019-librgw2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:41.822 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:41.913 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T12:18:41.931 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-16T12:18:41.971 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T12:18:42.382 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-16T12:18:42.402 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-16T12:18:42.465 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T12:18:42.608 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T12:18:42.625 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:42.658 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:42.843 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package librgw2. 2024-09-16T12:18:42.859 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../019-librgw2_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:42.901 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:43.019 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package liboath0. 2024-09-16T12:18:43.036 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-16T12:18:43.077 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-16T12:18:43.380 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T12:18:43.396 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:43.438 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:43.671 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T12:18:43.689 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:43.720 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:43.917 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-common. 2024-09-16T12:18:43.933 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:44.008 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:44.131 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liboath0. 2024-09-16T12:18:44.147 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-16T12:18:44.189 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-16T12:18:44.534 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T12:18:44.550 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:44.592 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:45.321 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-common. 2024-09-16T12:18:45.337 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:45.379 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:46.572 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-base. 2024-09-16T12:18:46.589 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:46.640 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:47.518 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-cryptography. 2024-09-16T12:18:47.535 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-16T12:18:47.576 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T12:18:47.947 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T12:18:47.967 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-16T12:18:48.005 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-16T12:18:48.017 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-base. 2024-09-16T12:18:48.035 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:48.085 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:48.291 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-packaging. 2024-09-16T12:18:48.310 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-16T12:18:48.349 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-16T12:18:48.602 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T12:18:48.621 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T12:18:48.660 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:48.971 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cryptography. 2024-09-16T12:18:48.991 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-16T12:18:49.032 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T12:18:49.048 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-16T12:18:49.063 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T12:18:49.084 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T12:18:49.384 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T12:18:49.400 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-16T12:18:49.442 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-16T12:18:49.458 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T12:18:49.474 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-16T12:18:49.516 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-16T12:18:49.796 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T12:18:49.812 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-16T12:18:49.844 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-packaging. 2024-09-16T12:18:49.854 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-16T12:18:49.861 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-16T12:18:49.902 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-16T12:18:50.146 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T12:18:50.165 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T12:18:50.165 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-jwt. 2024-09-16T12:18:50.184 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-16T12:18:50.196 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:50.232 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T12:18:50.527 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T12:18:50.546 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-16T12:18:50.585 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-16T12:18:50.592 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T12:18:50.609 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-16T12:18:50.667 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T12:18:50.888 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T12:18:50.904 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-16T12:18:50.946 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-16T12:18:51.012 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T12:18:51.029 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-16T12:18:51.070 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-16T12:18:51.232 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-mako. 2024-09-16T12:18:51.249 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-16T12:18:51.299 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T12:18:51.415 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T12:18:51.431 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-16T12:18:51.473 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-16T12:18:51.619 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T12:18:51.636 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T12:18:51.677 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T12:18:51.826 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jwt. 2024-09-16T12:18:51.844 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-16T12:18:51.909 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T12:18:51.980 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T12:18:51.998 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-16T12:18:52.038 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T12:18:52.262 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T12:18:52.278 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-16T12:18:52.321 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-16T12:18:52.349 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-webob. 2024-09-16T12:18:52.367 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-16T12:18:52.399 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T12:18:52.648 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T12:18:52.664 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-16T12:18:52.685 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-16T12:18:52.701 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-16T12:18:52.706 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-16T12:18:52.744 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T12:18:53.034 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-mako. 2024-09-16T12:18:53.046 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-bs4. 2024-09-16T12:18:53.054 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-16T12:18:53.066 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-16T12:18:53.092 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T12:18:53.104 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-16T12:18:53.448 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T12:18:53.455 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T12:18:53.463 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T12:18:53.472 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-16T12:18:53.516 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T12:18:53.539 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T12:18:53.818 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T12:18:53.835 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-16T12:18:53.867 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T12:18:53.875 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-16T12:18:53.886 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-16T12:18:53.925 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T12:18:54.178 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-paste. 2024-09-16T12:18:54.196 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-16T12:18:54.261 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-webob. 2024-09-16T12:18:54.261 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T12:18:54.277 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-16T12:18:54.319 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T12:18:54.582 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T12:18:54.600 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-16T12:18:54.639 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T12:18:54.664 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-16T12:18:54.683 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-16T12:18:54.722 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T12:18:54.943 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T12:18:54.960 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-16T12:18:55.001 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-16T12:18:55.054 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-bs4. 2024-09-16T12:18:55.072 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-16T12:18:55.129 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-16T12:18:55.287 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T12:18:55.304 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-16T12:18:55.345 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-16T12:18:55.558 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T12:18:55.576 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-16T12:18:55.619 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T12:18:55.657 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T12:18:55.673 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-16T12:18:55.715 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T12:18:55.986 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T12:18:56.001 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-16T12:18:56.062 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-16T12:18:56.062 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T12:18:56.079 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-16T12:18:56.120 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T12:18:56.397 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-paste. 2024-09-16T12:18:56.415 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-16T12:18:56.455 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T12:18:56.477 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T12:18:56.493 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:56.568 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:56.766 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T12:18:56.782 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-16T12:18:56.824 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T12:18:56.922 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T12:18:56.937 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:56.979 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:57.152 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T12:18:57.169 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-16T12:18:57.202 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-16T12:18:57.482 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T12:18:57.502 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:57.538 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T12:18:57.540 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:57.557 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-16T12:18:57.596 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-16T12:18:57.941 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T12:18:57.957 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-16T12:18:58.024 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T12:18:58.327 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T12:18:58.346 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:58.385 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:58.408 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T12:18:58.425 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-16T12:18:58.466 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T12:18:58.834 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T12:18:58.850 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:58.892 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:59.295 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T12:18:59.315 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:59.353 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:18:59.924 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T12:18:59.943 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:18:59.981 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:00.705 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph. 2024-09-16T12:19:00.723 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../053-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:00.763 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:01.083 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T12:19:01.100 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:01.166 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:01.585 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T12:19:01.595 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T12:19:01.601 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:01.612 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:01.652 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:01.668 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:02.189 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package cephadm. 2024-09-16T12:19:02.207 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../056-cephadm_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:02.247 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:02.542 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T12:19:02.557 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-16T12:19:02.591 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T12:19:02.878 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T12:19:02.896 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T12:19:02.936 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:03.256 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T12:19:03.272 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T12:19:03.305 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T12:19:03.566 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libjs-underscore. 2024-09-16T12:19:03.582 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-16T12:19:03.625 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T12:19:03.839 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph. 2024-09-16T12:19:03.857 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../053-ceph_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:03.887 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-16T12:19:03.888 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:03.904 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-16T12:19:03.945 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T12:19:04.217 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T12:19:04.233 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:04.282 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-routes. 2024-09-16T12:19:04.298 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-16T12:19:04.299 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:04.339 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-16T12:19:04.668 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T12:19:04.684 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T12:19:04.726 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:04.752 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T12:19:04.771 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:04.810 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:05.372 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package cephadm. 2024-09-16T12:19:05.389 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../056-cephadm_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:05.430 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:05.733 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T12:19:05.749 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-16T12:19:05.791 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T12:19:06.048 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-decorator. 2024-09-16T12:19:06.065 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-16T12:19:06.106 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T12:19:06.202 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T12:19:06.218 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T12:19:06.385 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:06.409 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-16T12:19:06.425 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-16T12:19:06.467 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T12:19:06.788 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-scipy. 2024-09-16T12:19:06.807 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-16T12:19:06.846 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-16T12:19:06.971 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T12:19:06.989 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T12:19:07.046 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T12:19:07.326 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libjs-underscore. 2024-09-16T12:19:07.344 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-16T12:19:07.390 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T12:19:07.693 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-16T12:19:07.709 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-16T12:19:07.751 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T12:19:08.103 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-routes. 2024-09-16T12:19:08.119 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-16T12:19:08.138 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-16T12:19:08.153 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-16T12:19:08.156 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-16T12:19:08.196 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T12:19:08.531 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T12:19:08.549 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T12:19:08.613 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:08.716 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T12:19:08.732 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-16T12:19:08.775 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-16T12:19:09.137 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T12:19:09.155 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-16T12:19:09.186 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T12:19:09.878 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T12:19:09.898 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T12:19:09.936 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:09.996 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-decorator. 2024-09-16T12:19:10.013 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-16T12:19:10.054 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T12:19:10.373 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-16T12:19:10.392 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-16T12:19:10.431 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T12:19:10.801 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-scipy. 2024-09-16T12:19:10.817 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-16T12:19:10.884 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-16T12:19:11.022 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T12:19:11.042 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-16T12:19:11.080 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-16T12:19:11.383 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T12:19:11.400 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-16T12:19:11.433 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T12:19:11.736 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T12:19:11.753 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-16T12:19:11.794 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T12:19:12.097 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T12:19:12.113 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-16T12:19:12.155 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-16T12:19:12.234 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-16T12:19:12.254 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-16T12:19:12.292 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T12:19:12.467 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T12:19:12.484 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-16T12:19:12.525 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-16T12:19:12.862 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T12:19:12.870 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-blinker. 2024-09-16T12:19:12.880 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-16T12:19:12.890 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-16T12:19:12.920 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-16T12:19:12.928 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T12:19:13.223 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-16T12:19:13.243 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-16T12:19:13.281 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T12:19:13.324 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T12:19:13.341 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-16T12:19:13.382 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T12:19:13.610 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T12:19:13.626 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-16T12:19:13.667 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T12:19:13.971 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T12:19:13.988 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-16T12:19:14.029 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T12:19:14.165 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T12:19:14.182 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T12:19:14.223 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:14.332 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T12:19:14.349 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-16T12:19:14.390 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T12:19:15.031 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T12:19:15.049 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T12:19:15.097 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:15.396 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T12:19:15.409 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T12:19:15.415 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-16T12:19:15.428 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-16T12:19:15.467 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-16T12:19:15.473 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-16T12:19:15.795 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T12:19:15.811 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-16T12:19:15.853 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T12:19:16.099 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T12:19:16.117 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T12:19:16.149 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T12:19:16.240 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T12:19:16.260 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-16T12:19:16.298 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T12:19:16.418 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package jq. 2024-09-16T12:19:16.438 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T12:19:16.476 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T12:19:16.634 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T12:19:16.650 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-16T12:19:16.717 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-16T12:19:16.746 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package socat. 2024-09-16T12:19:16.766 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-16T12:19:16.804 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-16T12:19:17.021 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T12:19:17.040 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-16T12:19:17.070 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-16T12:19:17.125 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T12:19:17.142 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-16T12:19:17.183 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-16T12:19:17.366 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-blinker. 2024-09-16T12:19:17.382 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-16T12:19:17.424 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T12:19:17.538 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-test. 2024-09-16T12:19:17.555 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:17.596 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:17.751 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-16T12:19:17.768 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-16T12:19:17.809 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T12:19:18.164 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T12:19:18.181 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-16T12:19:18.221 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T12:19:18.566 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T12:19:18.582 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-16T12:19:18.641 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T12:19:18.977 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T12:19:18.994 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-16T12:19:19.035 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T12:19:19.704 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T12:19:19.720 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T12:19:19.762 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:20.123 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T12:19:20.140 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-16T12:19:20.181 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-16T12:19:20.576 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T12:19:20.596 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T12:19:20.634 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T12:19:20.929 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package jq. 2024-09-16T12:19:20.945 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-16T12:19:20.987 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T12:19:21.265 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package socat. 2024-09-16T12:19:21.282 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-16T12:19:21.323 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-16T12:19:21.677 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T12:19:21.694 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-16T12:19:21.738 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-16T12:19:22.094 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-test. 2024-09-16T12:19:22.111 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:22.185 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:22.196 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T12:19:22.213 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T12:19:22.245 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:22.569 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-16T12:19:22.585 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T12:19:22.627 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T12:19:23.148 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T12:19:23.165 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:23.206 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:23.601 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T12:19:23.618 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-16T12:19:23.677 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T12:19:24.022 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-16T12:19:24.040 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T12:19:24.081 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T12:19:25.700 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-16T12:19:25.719 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-16T12:19:25.758 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-16T12:19:26.095 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-16T12:19:26.115 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-16T12:19:26.144 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T12:19:26.447 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-attr. 2024-09-16T12:19:26.465 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-16T12:19:26.530 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-16T12:19:26.819 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T12:19:26.840 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1643-gd75f2382-1focal_all.deb ... 2024-09-16T12:19:26.858 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-webencodings. 2024-09-16T12:19:26.875 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-16T12:19:26.888 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:26.916 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T12:19:27.219 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-html5lib. 2024-09-16T12:19:27.236 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-16T12:19:27.277 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-16T12:19:27.328 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-16T12:19:27.348 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T12:19:27.386 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T12:19:27.613 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-16T12:19:27.630 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-16T12:19:27.671 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T12:19:27.965 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T12:19:27.982 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:28.031 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:28.116 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T12:19:28.134 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T12:19:28.174 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T12:19:28.493 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T12:19:28.494 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T12:19:28.510 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-16T12:19:28.511 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-16T12:19:28.552 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-16T12:19:28.568 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T12:19:28.872 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T12:19:28.890 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-16T12:19:28.922 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-16T12:19:28.930 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T12:19:28.941 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-16T12:19:28.972 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T12:19:29.233 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-py. 2024-09-16T12:19:29.250 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-16T12:19:29.291 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T12:19:29.586 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T12:19:29.607 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-16T12:19:29.644 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T12:19:29.947 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T12:19:29.965 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-16T12:19:30.005 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T12:19:30.291 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T12:19:30.312 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-16T12:19:30.349 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-16T12:19:30.686 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T12:19:30.705 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-16T12:19:30.744 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T12:19:30.785 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-16T12:19:30.803 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-16T12:19:30.843 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-16T12:19:31.022 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T12:19:31.041 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-16T12:19:31.098 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T12:19:31.196 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-16T12:19:31.214 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-16T12:19:31.254 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T12:19:31.582 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-attr. 2024-09-16T12:19:31.599 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-16T12:19:31.639 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-16T12:19:31.793 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package radosgw. 2024-09-16T12:19:31.810 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../108-radosgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:31.850 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:32.017 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-webencodings. 2024-09-16T12:19:32.037 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-16T12:19:32.075 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T12:19:32.412 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-html5lib. 2024-09-16T12:19:32.430 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-16T12:19:32.470 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-16T12:19:32.840 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-16T12:19:32.860 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-16T12:19:32.898 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T12:19:33.393 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T12:19:33.410 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T12:19:33.450 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T12:19:33.609 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T12:19:33.627 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:33.667 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:33.804 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T12:19:33.822 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-16T12:19:33.862 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-16T12:19:34.011 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package rbd-mirror. 2024-09-16T12:19:34.029 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../110-rbd-mirror_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:34.069 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking rbd-mirror (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:34.199 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T12:19:34.215 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-16T12:19:34.281 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T12:19:34.651 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-py. 2024-09-16T12:19:34.667 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-16T12:19:34.672 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package rbd-nbd. 2024-09-16T12:19:34.689 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../111-rbd-nbd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:34.709 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T12:19:34.730 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking rbd-nbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:35.045 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-16T12:19:35.054 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T12:19:35.071 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-16T12:19:35.112 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T12:19:35.360 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T12:19:35.440 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T12:19:35.458 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-16T12:19:35.486 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T12:19:35.521 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T12:19:35.595 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T12:19:35.824 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:35.851 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T12:19:35.869 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-16T12:19:35.909 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-16T12:19:36.297 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T12:19:36.317 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-16T12:19:36.357 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T12:19:36.385 INFO:teuthology.orchestra.run.smithi018.stdout:Adding system user cephadm....done 2024-09-16T12:19:36.538 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T12:19:36.664 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T12:19:36.668 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T12:19:36.686 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-16T12:19:36.726 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T12:19:37.024 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T12:19:37.325 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T12:19:37.452 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-16T12:19:37.553 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T12:19:37.839 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package radosgw. 2024-09-16T12:19:37.856 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../108-radosgw_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:37.888 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-16T12:19:37.922 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:38.307 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T12:19:38.608 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-16T12:19:38.917 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T12:19:39.219 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T12:19:39.519 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:39.675 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T12:19:39.692 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:39.733 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:39.804 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T12:19:39.929 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-16T12:19:40.112 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package rbd-mirror. 2024-09-16T12:19:40.133 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../110-rbd-mirror_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:40.194 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking rbd-mirror (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:40.214 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T12:19:40.532 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T12:19:40.666 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T12:19:40.799 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package rbd-nbd. 2024-09-16T12:19:40.818 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../111-rbd-nbd_17.2.7-1643-gd75f2382-1focal_amd64.deb ... 2024-09-16T12:19:40.882 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking rbd-nbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:41.050 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-16T12:19:41.230 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-16T12:19:41.352 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T12:19:41.546 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-16T12:19:41.672 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-16T12:19:41.677 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T12:19:41.798 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-16T12:19:42.020 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T12:19:42.024 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:42.305 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T12:19:42.620 INFO:teuthology.orchestra.run.smithi172.stdout:Adding system user cephadm....done 2024-09-16T12:19:42.655 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-16T12:19:42.807 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-16T12:19:42.933 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-16T12:19:42.973 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-16T12:19:43.285 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T12:19:43.291 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T12:19:43.417 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T12:19:43.543 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T12:19:43.586 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-16T12:19:43.712 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-16T12:19:43.844 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T12:19:43.863 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-16T12:19:44.206 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-16T12:19:44.246 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T12:19:44.608 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-16T12:19:44.614 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T12:19:44.934 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-16T12:19:44.990 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-16T12:19:45.125 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:45.244 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-16T12:19:45.459 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T12:19:45.562 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-16T12:19:45.593 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-16T12:19:45.888 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-ceph-argparse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:45.962 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T12:19:46.071 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-16T12:19:46.214 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-16T12:19:46.313 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-16T12:19:46.365 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-16T12:19:46.657 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-16T12:19:46.958 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-16T12:19:46.973 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T12:19:47.067 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-16T12:19:47.299 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T12:19:47.425 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T12:19:47.452 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-16T12:19:47.551 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T12:19:47.845 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-16T12:19:47.886 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T12:19:48.187 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-16T12:19:48.261 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-16T12:19:48.471 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T12:19:48.597 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T12:19:48.614 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-16T12:19:48.890 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-16T12:19:48.923 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-16T12:19:49.207 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T12:19:49.258 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-16T12:19:49.533 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T12:19:49.592 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-16T12:19:49.910 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-16T12:19:49.912 INFO:teuthology.orchestra.run.smithi018.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-16T12:19:50.036 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-16T12:19:50.161 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-16T12:19:50.187 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-16T12:19:50.287 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T12:19:50.513 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-16T12:19:50.588 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-16T12:19:50.915 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-16T12:19:50.972 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T12:19:51.107 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T12:19:51.233 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T12:19:51.283 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-16T12:19:51.584 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T12:19:51.643 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-16T12:19:51.811 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:51.893 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up librados2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:52.019 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-16T12:19:52.145 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T12:19:52.279 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-16T12:19:52.623 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-16T12:19:52.782 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-16T12:19:52.908 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-16T12:19:53.201 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-16T12:19:53.527 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T12:19:53.549 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:53.653 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-16T12:19:53.675 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T12:19:53.801 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-16T12:19:53.812 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-16T12:19:54.122 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-16T12:19:54.203 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-16T12:19:54.398 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-16T12:19:54.504 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-16T12:19:54.699 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-16T12:19:54.797 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T12:19:54.825 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T12:19:54.922 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-16T12:19:55.084 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-16T12:19:55.274 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:55.361 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-16T12:19:55.400 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T12:19:55.654 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-16T12:19:55.743 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:55.869 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T12:19:56.023 INFO:teuthology.orchestra.run.smithi172.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-16T12:19:56.220 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T12:19:56.298 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-16T12:19:56.423 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-16T12:19:56.750 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-16T12:19:57.118 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-16T12:19:57.269 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-16T12:19:57.292 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up librbd1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:57.395 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-16T12:19:57.418 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T12:19:57.544 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-16T12:19:57.670 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T12:19:57.696 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-16T12:19:57.788 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-16T12:19:57.980 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librados2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:58.072 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T12:19:58.106 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-16T12:19:58.198 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-16T12:19:58.516 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:58.641 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:58.907 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T12:19:58.907 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T12:19:59.376 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:59.570 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:19:59.578 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T12:19:59.696 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-16T12:19:59.847 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-16T12:19:59.880 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T12:20:00.181 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T12:20:00.215 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-16T12:20:00.500 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-16T12:20:00.509 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:00.635 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:00.761 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:00.784 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-16T12:20:00.886 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:00.918 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-16T12:20:01.012 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up rbd-nbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:01.155 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-16T12:20:01.245 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libcephfs2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:01.396 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-16T12:20:01.456 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-16T12:20:01.731 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libradosstriper1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:01.741 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:01.850 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:01.857 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-16T12:20:01.977 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T12:20:02.183 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-16T12:20:02.955 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:03.230 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librbd1 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:03.406 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-16T12:20:03.466 INFO:teuthology.orchestra.run.smithi018.stdout:Adding group ceph....done 2024-09-16T12:20:03.532 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-16T12:20:03.691 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-16T12:20:03.809 INFO:teuthology.orchestra.run.smithi018.stdout:Adding system user ceph....done 2024-09-16T12:20:03.834 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-16T12:20:03.985 INFO:teuthology.orchestra.run.smithi018.stdout:Setting system user ceph properties....done 2024-09-16T12:20:04.010 INFO:teuthology.orchestra.run.smithi018.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T12:20:04.158 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T12:20:04.179 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-16T12:20:04.330 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-16T12:20:04.488 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T12:20:04.639 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-modules-core (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:04.765 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:04.948 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:04.991 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T12:20:04.991 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T12:20:05.074 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:05.342 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T12:20:05.343 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T12:20:05.442 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up libcephfs-dev (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:05.687 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-16T12:20:05.980 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T12:20:06.001 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:06.306 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-16T12:20:06.390 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T12:20:06.675 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rados (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:06.843 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up librgw2 (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:06.869 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:06.969 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:07.103 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T12:20:07.103 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T12:20:07.104 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up rbd-fuse (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:07.255 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up rbd-nbd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:07.381 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-16T12:20:07.537 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up rbd-mirror (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:07.715 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-16T12:20:07.765 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T12:20:07.765 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T12:20:08.050 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-rgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:08.201 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-cephfs (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:08.238 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:08.319 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-16T12:20:08.467 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T12:20:08.467 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T12:20:08.907 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:09.278 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-common (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:09.391 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T12:20:09.391 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T12:20:09.850 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:09.883 INFO:teuthology.orchestra.run.smithi172.stdout:Adding group ceph....done 2024-09-16T12:20:09.976 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:10.118 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:10.293 INFO:teuthology.orchestra.run.smithi172.stdout:Adding system user ceph....done 2024-09-16T12:20:10.343 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T12:20:10.343 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T12:20:10.503 INFO:teuthology.orchestra.run.smithi172.stdout:Setting system user ceph properties....done 2024-09-16T12:20:10.523 INFO:teuthology.orchestra.run.smithi172.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T12:20:10.672 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T12:20:10.803 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:10.946 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:10.987 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T12:20:11.071 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:11.214 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:11.333 INFO:teuthology.orchestra.run.smithi018.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-16T12:20:11.449 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-test (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:11.485 INFO:teuthology.orchestra.run.smithi018.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-16T12:20:11.574 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up radosgw (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:11.769 INFO:teuthology.orchestra.run.smithi018.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-16T12:20:11.863 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T12:20:11.863 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T12:20:12.535 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-base (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:12.946 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T12:20:13.412 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mds (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:13.640 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T12:20:13.640 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T12:20:14.080 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up rbd-mirror (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:14.337 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T12:20:14.337 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-09-16T12:20:14.814 INFO:teuthology.orchestra.run.smithi018.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-16T12:20:14.817 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:15.048 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T12:20:15.048 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T12:20:15.470 INFO:teuthology.orchestra.run.smithi018.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T12:20:15.472 DEBUG:teuthology.orchestra.run.smithi018:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install pv 2024-09-16T12:20:15.507 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-osd (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:15.550 INFO:teuthology.orchestra.run.smithi018.stdout:Reading package lists... 2024-09-16T12:20:15.776 INFO:teuthology.orchestra.run.smithi018.stdout:Building dependency tree... 2024-09-16T12:20:15.777 INFO:teuthology.orchestra.run.smithi018.stdout:Reading state information... 2024-09-16T12:20:15.990 INFO:teuthology.orchestra.run.smithi018.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T12:20:15.990 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T12:20:15.990 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T12:20:15.991 INFO:teuthology.orchestra.run.smithi018.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-16T12:20:15.991 INFO:teuthology.orchestra.run.smithi018.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T12:20:15.992 INFO:teuthology.orchestra.run.smithi018.stdout:Suggested packages: 2024-09-16T12:20:15.992 INFO:teuthology.orchestra.run.smithi018.stdout: doc-base 2024-09-16T12:20:16.025 INFO:teuthology.orchestra.run.smithi018.stdout:The following NEW packages will be installed: 2024-09-16T12:20:16.025 INFO:teuthology.orchestra.run.smithi018.stdout: pv 2024-09-16T12:20:16.208 INFO:teuthology.orchestra.run.smithi018.stdout:0 upgraded, 1 newly installed, 0 to remove and 295 not upgraded. 2024-09-16T12:20:16.208 INFO:teuthology.orchestra.run.smithi018.stdout:Need to get 48.3 kB of archives. 2024-09-16T12:20:16.208 INFO:teuthology.orchestra.run.smithi018.stdout:After this operation, 123 kB of additional disk space will be used. 2024-09-16T12:20:16.209 INFO:teuthology.orchestra.run.smithi018.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 pv amd64 1.6.6-1 [48.3 kB] 2024-09-16T12:20:16.459 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:16.585 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:16.748 INFO:teuthology.orchestra.run.smithi018.stdout:Fetched 48.3 kB in 0s (142 kB/s) 2024-09-16T12:20:16.770 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mon (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:16.835 INFO:teuthology.orchestra.run.smithi018.stdout:Selecting previously unselected package pv. 2024-09-16T12:20:16.892 INFO:teuthology.orchestra.run.smithi018.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 152924 files and directories currently installed.) 2024-09-16T12:20:16.896 INFO:teuthology.orchestra.run.smithi018.stdout:Preparing to unpack .../archives/pv_1.6.6-1_amd64.deb ... 2024-09-16T12:20:16.941 INFO:teuthology.orchestra.run.smithi018.stdout:Unpacking pv (1.6.6-1) ... 2024-09-16T12:20:16.991 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T12:20:16.991 INFO:teuthology.orchestra.run.smithi172.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T12:20:17.310 INFO:teuthology.orchestra.run.smithi018.stdout:Setting up pv (1.6.6-1) ... 2024-09-16T12:20:17.438 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-cephadm (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:17.556 INFO:teuthology.orchestra.run.smithi018.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-16T12:20:17.580 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:17.740 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-mgr-dashboard (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:17.882 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up ceph-volume (17.2.7-1643-gd75f2382-1focal) ... 2024-09-16T12:20:18.043 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-16T12:20:18.194 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-16T12:20:18.457 INFO:teuthology.orchestra.run.smithi018.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T12:20:18.463 DEBUG:teuthology.parallel:result is None 2024-09-16T12:20:18.504 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-16T12:20:21.366 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-16T12:20:22.003 INFO:teuthology.orchestra.run.smithi172.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T12:20:22.005 DEBUG:teuthology.orchestra.run.smithi172:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install pv 2024-09-16T12:20:22.076 INFO:teuthology.orchestra.run.smithi172.stdout:Reading package lists... 2024-09-16T12:20:22.293 INFO:teuthology.orchestra.run.smithi172.stdout:Building dependency tree... 2024-09-16T12:20:22.294 INFO:teuthology.orchestra.run.smithi172.stdout:Reading state information... 2024-09-16T12:20:22.499 INFO:teuthology.orchestra.run.smithi172.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T12:20:22.500 INFO:teuthology.orchestra.run.smithi172.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-16T12:20:22.500 INFO:teuthology.orchestra.run.smithi172.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T12:20:22.501 INFO:teuthology.orchestra.run.smithi172.stdout:Suggested packages: 2024-09-16T12:20:22.501 INFO:teuthology.orchestra.run.smithi172.stdout: doc-base 2024-09-16T12:20:22.534 INFO:teuthology.orchestra.run.smithi172.stdout:The following NEW packages will be installed: 2024-09-16T12:20:22.534 INFO:teuthology.orchestra.run.smithi172.stdout: pv 2024-09-16T12:20:22.590 INFO:teuthology.orchestra.run.smithi172.stdout:0 upgraded, 1 newly installed, 0 to remove and 295 not upgraded. 2024-09-16T12:20:22.590 INFO:teuthology.orchestra.run.smithi172.stdout:Need to get 48.3 kB of archives. 2024-09-16T12:20:22.590 INFO:teuthology.orchestra.run.smithi172.stdout:After this operation, 123 kB of additional disk space will be used. 2024-09-16T12:20:22.590 INFO:teuthology.orchestra.run.smithi172.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 pv amd64 1.6.6-1 [48.3 kB] 2024-09-16T12:20:22.981 INFO:teuthology.orchestra.run.smithi172.stdout:Fetched 48.3 kB in 0s (598 kB/s) 2024-09-16T12:20:23.070 INFO:teuthology.orchestra.run.smithi172.stdout:Selecting previously unselected package pv. 2024-09-16T12:20:23.129 INFO:teuthology.orchestra.run.smithi172.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 152924 files and directories currently installed.) 2024-09-16T12:20:23.133 INFO:teuthology.orchestra.run.smithi172.stdout:Preparing to unpack .../archives/pv_1.6.6-1_amd64.deb ... 2024-09-16T12:20:23.176 INFO:teuthology.orchestra.run.smithi172.stdout:Unpacking pv (1.6.6-1) ... 2024-09-16T12:20:23.538 INFO:teuthology.orchestra.run.smithi172.stdout:Setting up pv (1.6.6-1) ... 2024-09-16T12:20:23.665 INFO:teuthology.orchestra.run.smithi172.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-16T12:20:24.478 INFO:teuthology.orchestra.run.smithi172.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T12:20:24.482 DEBUG:teuthology.parallel:result is None 2024-09-16T12:20:24.483 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T12:20:24.641 DEBUG:teuthology.orchestra.run.smithi018:> dpkg-query -W -f '${Version}' ceph 2024-09-16T12:20:24.664 INFO:teuthology.orchestra.run.smithi018.stdout:17.2.7-1643-gd75f2382-1focal 2024-09-16T12:20:24.665 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643-gd75f2382-1focal 2024-09-16T12:20:24.665 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643-gd75f2382-1focal is installed. 2024-09-16T12:20:24.667 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-16T12:20:24.821 DEBUG:teuthology.orchestra.run.smithi172:> dpkg-query -W -f '${Version}' ceph 2024-09-16T12:20:24.844 INFO:teuthology.orchestra.run.smithi172.stdout:17.2.7-1643-gd75f2382-1focal 2024-09-16T12:20:24.844 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643-gd75f2382-1focal 2024-09-16T12:20:24.845 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643-gd75f2382-1focal is installed. 2024-09-16T12:20:24.846 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T12:20:24.847 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:20:24.847 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T12:20:24.863 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:20:24.863 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T12:20:24.902 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T12:20:24.903 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:20:24.903 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T12:20:24.919 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T12:20:24.977 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:20:24.977 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T12:20:24.993 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T12:20:25.007 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T12:20:25.008 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:20:25.008 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T12:20:25.024 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T12:20:25.035 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:20:25.036 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T12:20:25.066 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T12:20:25.077 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T12:20:25.077 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:20:25.077 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T12:20:25.094 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T12:20:25.153 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:20:25.153 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T12:20:25.169 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T12:20:25.225 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T12:20:25.303 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-16T12:20:25.303 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 777 /var/log/ceph 2024-09-16T12:20:25.305 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /var/log/ceph 2024-09-16T12:20:25.321 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-16T12:20:25.321 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T12:20:25.364 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-16T12:20:25.379 INFO:tasks.ceph:Creating extra log directories... 2024-09-16T12:20:25.379 DEBUG:teuthology.orchestra.run.smithi018:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T12:20:25.381 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-16T12:20:25.450 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-09-16T12:20:25.450 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T12:20:25.451 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907248', 'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/compare-mirror-images-nbd}', 'email': 'sangadi@redhat.com', 'first_in_suite': False, 'job_id': '7907248', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}], 'name': 'sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'extra_system_packages': ['pv']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'owner': 'scheduled_sangadi@teuthology', 'priority': 130, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9764, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'targets': {'smithi018.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi172.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror', 'rbd-nbd'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'extra_system_packages': ['pv']}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/compare_mirror_images.sh']}, 'env': {'RBD_DEVICE_TYPE': 'nbd'}, 'timeout': '3h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-16_10:44:09', 'tube': 'smithi', 'user': 'sangadi', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T12:20:25.451 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-16T12:20:25.453 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-16T12:20:25.460 DEBUG:teuthology.orchestra.run.smithi018:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T12:20:25.462 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T12:20:25.475 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:20:25.475 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T12:20:25.524 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T12:20:25.525 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_1 2024-09-16T12:20:25.532 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T12:20:25.532 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T12:20:25.532 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 6h/6d Inode: 359 Links: 1 2024-09-16T12:20:25.532 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T12:20:25.532 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2024-09-16 12:16:47.842455034 +0000 2024-09-16T12:20:25.532 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2024-09-16 12:16:47.634459824 +0000 2024-09-16T12:20:25.533 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2024-09-16 12:16:47.634459824 +0000 2024-09-16T12:20:25.533 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: - 2024-09-16T12:20:25.533 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T12:20:25.547 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2024-09-16T12:20:25.547 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2024-09-16T12:20:25.547 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 0.000308526 s, 1.7 MB/s 2024-09-16T12:20:25.548 DEBUG:teuthology.orchestra.run.smithi018:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T12:20:25.556 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_2 2024-09-16T12:20:25.564 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T12:20:25.564 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T12:20:25.564 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 6h/6d Inode: 365 Links: 1 2024-09-16T12:20:25.564 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T12:20:25.564 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2024-09-16 12:16:48.250445639 +0000 2024-09-16T12:20:25.564 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2024-09-16 12:16:48.038450521 +0000 2024-09-16T12:20:25.564 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2024-09-16 12:16:48.038450521 +0000 2024-09-16T12:20:25.564 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: - 2024-09-16T12:20:25.564 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T12:20:25.574 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2024-09-16T12:20:25.574 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2024-09-16T12:20:25.575 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 0.000218627 s, 2.3 MB/s 2024-09-16T12:20:25.575 DEBUG:teuthology.orchestra.run.smithi018:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T12:20:25.626 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_3 2024-09-16T12:20:25.634 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T12:20:25.634 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T12:20:25.634 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 6h/6d Inode: 373 Links: 1 2024-09-16T12:20:25.634 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T12:20:25.634 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2024-09-16 12:16:48.662436152 +0000 2024-09-16T12:20:25.634 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2024-09-16 12:16:48.466440665 +0000 2024-09-16T12:20:25.634 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2024-09-16 12:16:48.466440665 +0000 2024-09-16T12:20:25.634 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: - 2024-09-16T12:20:25.634 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T12:20:25.649 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2024-09-16T12:20:25.649 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2024-09-16T12:20:25.649 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 0.000239057 s, 2.1 MB/s 2024-09-16T12:20:25.650 DEBUG:teuthology.orchestra.run.smithi018:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T12:20:25.658 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_4 2024-09-16T12:20:25.663 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T12:20:25.663 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T12:20:25.664 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 6h/6d Inode: 377 Links: 1 2024-09-16T12:20:25.664 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T12:20:25.664 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2024-09-16 12:16:49.150424914 +0000 2024-09-16T12:20:25.664 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2024-09-16 12:16:48.894430809 +0000 2024-09-16T12:20:25.664 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2024-09-16 12:16:48.894430809 +0000 2024-09-16T12:20:25.664 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: - 2024-09-16T12:20:25.664 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T12:20:25.720 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2024-09-16T12:20:25.720 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2024-09-16T12:20:25.720 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 0.000289566 s, 1.8 MB/s 2024-09-16T12:20:25.721 DEBUG:teuthology.orchestra.run.smithi018:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T12:20:25.729 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T12:20:25.730 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi018.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T12:20:25.730 INFO:tasks.ceph:Generating config... 2024-09-16T12:20:25.731 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-16T12:20:25.731 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-16T12:20:25.732 INFO:tasks.ceph:[client] log to stderr = False 2024-09-16T12:20:25.732 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-16T12:20:25.732 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:25.732 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:25.732 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:25.732 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:25.732 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:25.732 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:25.732 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:25.732 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:25.732 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:25.732 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:25.733 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:25.733 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:25.733 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:25.733 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:25.733 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T12:20:25.733 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-16T12:20:25.733 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T12:20:25.733 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T12:20:25.733 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T12:20:25.733 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T12:20:25.733 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T12:20:25.733 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T12:20:25.733 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-09-16T12:20:25.734 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-09-16T12:20:25.734 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T12:20:25.734 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T12:20:25.734 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-16T12:20:25.734 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-16T12:20:25.734 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T12:20:25.734 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T12:20:25.734 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-16T12:20:25.734 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T12:20:25.734 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T12:20:25.734 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T12:20:25.734 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T12:20:25.734 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T12:20:25.735 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-09-16T12:20:25.735 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-09-16T12:20:25.764 INFO:teuthology.orchestra.run.smithi018.stdout:creating /etc/ceph/cluster1.keyring 2024-09-16T12:20:25.767 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-09-16T12:20:25.800 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-16T12:20:25.857 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.18')] 2024-09-16T12:20:25.858 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.18', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-16T12:20:25.858 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:20:25.858 DEBUG:teuthology.orchestra.run.smithi018:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T12:20:25.864 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.18 --print /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T12:20:25.897 INFO:teuthology.orchestra.run.smithi018.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T12:20:25.897 INFO:teuthology.orchestra.run.smithi018.stdout:monmaptool: generated fsid 3d80de80-eeb4-495e-a052-ec9a35b35bbd 2024-09-16T12:20:25.897 INFO:teuthology.orchestra.run.smithi018.stdout:setting min_mon_release = octopus 2024-09-16T12:20:25.897 INFO:teuthology.orchestra.run.smithi018.stdout:epoch 0 2024-09-16T12:20:25.897 INFO:teuthology.orchestra.run.smithi018.stdout:fsid 3d80de80-eeb4-495e-a052-ec9a35b35bbd 2024-09-16T12:20:25.897 INFO:teuthology.orchestra.run.smithi018.stdout:last_changed 2024-09-16T12:20:25.893664+0000 2024-09-16T12:20:25.897 INFO:teuthology.orchestra.run.smithi018.stdout:created 2024-09-16T12:20:25.893664+0000 2024-09-16T12:20:25.897 INFO:teuthology.orchestra.run.smithi018.stdout:min_mon_release 15 (octopus) 2024-09-16T12:20:25.898 INFO:teuthology.orchestra.run.smithi018.stdout:election_strategy: 1 2024-09-16T12:20:25.898 INFO:teuthology.orchestra.run.smithi018.stdout:0: [v2:172.21.15.18:3300/0,v1:172.21.15.18:6789/0] mon.a 2024-09-16T12:20:25.898 INFO:teuthology.orchestra.run.smithi018.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-09-16T12:20:25.899 DEBUG:teuthology.orchestra.run.smithi018:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T12:20:25.905 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 3d80de80-eeb4-495e-a052-ec9a35b35bbd... 2024-09-16T12:20:25.907 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-16T12:20:25.948 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-16T12:20:25.981 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2024-09-16T12:20:25.981 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2024-09-16T12:20:25.981 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T12:20:25.981 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = cephx 2024-09-16T12:20:25.981 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.981 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2024-09-16T12:20:25.981 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.981 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2024-09-16T12:20:25.981 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.982 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2024-09-16T12:20:25.982 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2024-09-16T12:20:25.982 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.982 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2024-09-16T12:20:25.982 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2024-09-16T12:20:25.982 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.982 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T12:20:25.982 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2024-09-16T12:20:25.982 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.982 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T12:20:25.982 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T12:20:25.982 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.982 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2024-09-16T12:20:25.983 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.983 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2024-09-16T12:20:25.983 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2024-09-16T12:20:25.983 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2024-09-16T12:20:25.983 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2024-09-16T12:20:25.983 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2024-09-16T12:20:25.983 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2024-09-16T12:20:25.983 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2024-09-16T12:20:25.983 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T12:20:25.983 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T12:20:25.983 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2024-09-16T12:20:25.983 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.983 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T12:20:25.984 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.984 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2024-09-16T12:20:25.984 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.984 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2024-09-16T12:20:25.984 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.984 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2024-09-16T12:20:25.984 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2024-09-16T12:20:25.984 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2024-09-16T12:20:25.984 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = 172.21.15.18 2024-09-16T12:20:25.984 INFO:teuthology.orchestra.run.smithi172.stdout: mon client directed command retry = 5 2024-09-16T12:20:25.984 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on pool no app = False 2024-09-16T12:20:25.984 INFO:teuthology.orchestra.run.smithi172.stdout: ms inject socket failures = 5000 2024-09-16T12:20:25.985 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = 3d80de80-eeb4-495e-a052-ec9a35b35bbd 2024-09-16T12:20:25.985 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.985 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2024-09-16T12:20:25.985 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2024-09-16T12:20:25.985 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.985 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2024-09-16T12:20:25.985 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 600 2024-09-16T12:20:25.985 INFO:teuthology.orchestra.run.smithi172.stdout: osd mclock profile = high_recovery_ops 2024-09-16T12:20:25.985 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.985 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2024-09-16T12:20:25.985 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2024-09-16T12:20:25.985 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.986 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2024-09-16T12:20:25.986 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2024-09-16T12:20:25.986 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2024-09-16T12:20:25.986 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.986 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2024-09-16T12:20:25.986 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2024-09-16T12:20:25.986 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.986 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2024-09-16T12:20:25.986 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.986 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2024-09-16T12:20:25.986 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.986 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2024-09-16T12:20:25.986 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.987 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2024-09-16T12:20:25.987 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2024-09-16T12:20:25.987 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.987 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2024-09-16T12:20:25.987 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2024-09-16T12:20:25.987 INFO:teuthology.orchestra.run.smithi172.stdout: bluefs allocator = hybrid 2024-09-16T12:20:25.987 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore allocator = hybrid 2024-09-16T12:20:25.987 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore block size = 96636764160 2024-09-16T12:20:25.987 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore fsck on mount = True 2024-09-16T12:20:25.987 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluefs = 1/20 2024-09-16T12:20:25.987 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluestore = 1/20 2024-09-16T12:20:25.987 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-09-16T12:20:25.987 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 20 2024-09-16T12:20:25.988 INFO:teuthology.orchestra.run.smithi172.stdout: debug rocksdb = 4/10 2024-09-16T12:20:25.988 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T12:20:25.988 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd full ratio = 0.9 2024-09-16T12:20:25.988 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd nearfull ratio = 0.8 2024-09-16T12:20:25.988 INFO:teuthology.orchestra.run.smithi172.stdout: osd failsafe full ratio = 0.95 2024-09-16T12:20:25.988 INFO:teuthology.orchestra.run.smithi172.stdout: osd objectstore = bluestore 2024-09-16T12:20:25.988 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.988 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2024-09-16T12:20:25.988 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-09-16T12:20:25.988 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2024-09-16T12:20:25.988 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2024-09-16T12:20:25.988 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2024-09-16T12:20:25.988 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2024-09-16T12:20:25.989 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2024-09-16T12:20:25.989 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2024-09-16T12:20:25.989 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.989 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2024-09-16T12:20:25.989 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-09-16T12:20:25.989 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2024-09-16T12:20:25.989 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 20 2024-09-16T12:20:25.989 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2024-09-16T12:20:25.989 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2024-09-16T12:20:25.989 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2024-09-16T12:20:25.989 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2024-09-16T12:20:25.989 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2024-09-16T12:20:25.989 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2024-09-16T12:20:25.990 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2024-09-16T12:20:25.990 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.990 INFO:teuthology.orchestra.run.smithi172.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T12:20:25.990 INFO:teuthology.orchestra.run.smithi172.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T12:20:25.990 INFO:teuthology.orchestra.run.smithi172.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T12:20:25.990 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.990 INFO:teuthology.orchestra.run.smithi172.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T12:20:25.990 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T12:20:25.990 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T12:20:25.990 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.990 INFO:teuthology.orchestra.run.smithi172.stdout: # 1m isn't quite enough 2024-09-16T12:20:25.990 INFO:teuthology.orchestra.run.smithi172.stdout: mon_down_mkfs_grace = 2m 2024-09-16T12:20:25.990 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.991 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_filestore_osds = false 2024-09-16T12:20:25.991 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:20:25.991 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2024-09-16T12:20:25.991 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2024-09-16T12:20:25.991 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2024-09-16T12:20:25.991 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2024-09-16T12:20:25.991 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T12:20:25.991 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T12:20:25.991 INFO:teuthology.orchestra.run.smithi172.stdout: debug rbd = 20 2024-09-16T12:20:25.991 INFO:teuthology.orchestra.run.smithi172.stdout: debug rbd_mirror = 15 2024-09-16T12:20:25.991 INFO:teuthology.orchestra.run.smithi172.stdout: log to stderr = False 2024-09-16T12:20:25.991 INFO:teuthology.orchestra.run.smithi172.stdout: rbd default features = 125 2024-09-16T12:20:25.991 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2024-09-16T12:20:25.991 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.0] 2024-09-16T12:20:25.992 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:25.992 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:25.992 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.1] 2024-09-16T12:20:25.992 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:25.992 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:25.992 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.2] 2024-09-16T12:20:25.992 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:25.992 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:25.992 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.3] 2024-09-16T12:20:25.992 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:25.992 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:25.992 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.4] 2024-09-16T12:20:25.992 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:25.992 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:25.993 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.5] 2024-09-16T12:20:25.993 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:25.993 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:25.993 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.6] 2024-09-16T12:20:25.993 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:25.993 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:25.994 INFO:teuthology.orchestra.run.smithi018.stdout:[global] 2024-09-16T12:20:25.994 INFO:teuthology.orchestra.run.smithi018.stdout: chdir = "" 2024-09-16T12:20:25.995 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T12:20:25.995 INFO:teuthology.orchestra.run.smithi018.stdout: auth supported = cephx 2024-09-16T12:20:25.995 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:25.995 INFO:teuthology.orchestra.run.smithi018.stdout: filestore xattr use omap = true 2024-09-16T12:20:25.995 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:25.995 INFO:teuthology.orchestra.run.smithi018.stdout: mon clock drift allowed = 1.000 2024-09-16T12:20:25.995 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:25.995 INFO:teuthology.orchestra.run.smithi018.stdout: osd crush chooseleaf type = 0 2024-09-16T12:20:25.995 INFO:teuthology.orchestra.run.smithi018.stdout: auth debug = true 2024-09-16T12:20:25.995 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:25.995 INFO:teuthology.orchestra.run.smithi018.stdout: ms die on old message = true 2024-09-16T12:20:25.996 INFO:teuthology.orchestra.run.smithi018.stdout: ms die on bug = true 2024-09-16T12:20:25.996 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:25.996 INFO:teuthology.orchestra.run.smithi018.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T12:20:25.996 INFO:teuthology.orchestra.run.smithi018.stdout: mon pg warn max object skew = 0 2024-09-16T12:20:25.996 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:25.996 INFO:teuthology.orchestra.run.smithi018.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T12:20:25.996 INFO:teuthology.orchestra.run.smithi018.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T12:20:25.996 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:25.996 INFO:teuthology.orchestra.run.smithi018.stdout: osd pool default size = 2 2024-09-16T12:20:25.996 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:25.996 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd allow primary affinity = true 2024-09-16T12:20:25.996 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd allow pg remap = true 2024-09-16T12:20:25.996 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on legacy crush tunables = false 2024-09-16T12:20:25.997 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on crush straw calc version zero = false 2024-09-16T12:20:25.997 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on no sortbitwise = false 2024-09-16T12:20:25.997 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on osd down out interval zero = false 2024-09-16T12:20:25.997 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on too few osds = false 2024-09-16T12:20:25.997 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T12:20:25.997 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T12:20:25.997 INFO:teuthology.orchestra.run.smithi018.stdout: mon_allow_pool_size_one = true 2024-09-16T12:20:25.997 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:25.997 INFO:teuthology.orchestra.run.smithi018.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T12:20:25.997 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:25.997 INFO:teuthology.orchestra.run.smithi018.stdout: osd default data pool replay window = 5 2024-09-16T12:20:25.998 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:25.998 INFO:teuthology.orchestra.run.smithi018.stdout: mon allow pool delete = true 2024-09-16T12:20:25.998 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:25.998 INFO:teuthology.orchestra.run.smithi018.stdout: mon cluster log file level = debug 2024-09-16T12:20:25.998 INFO:teuthology.orchestra.run.smithi018.stdout: debug asserts on shutdown = true 2024-09-16T12:20:25.998 INFO:teuthology.orchestra.run.smithi018.stdout: mon health detail to clog = false 2024-09-16T12:20:25.998 INFO:teuthology.orchestra.run.smithi018.stdout: mon host = 172.21.15.18 2024-09-16T12:20:25.998 INFO:teuthology.orchestra.run.smithi018.stdout: mon client directed command retry = 5 2024-09-16T12:20:25.998 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on pool no app = False 2024-09-16T12:20:25.998 INFO:teuthology.orchestra.run.smithi018.stdout: ms inject socket failures = 5000 2024-09-16T12:20:25.998 INFO:teuthology.orchestra.run.smithi018.stdout: fsid = 3d80de80-eeb4-495e-a052-ec9a35b35bbd 2024-09-16T12:20:25.998 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:25.998 INFO:teuthology.orchestra.run.smithi018.stdout:[osd] 2024-09-16T12:20:25.999 INFO:teuthology.orchestra.run.smithi018.stdout: osd journal size = 100 2024-09-16T12:20:25.999 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:25.999 INFO:teuthology.orchestra.run.smithi018.stdout: osd scrub load threshold = 5.0 2024-09-16T12:20:25.999 INFO:teuthology.orchestra.run.smithi018.stdout: osd scrub max interval = 600 2024-09-16T12:20:25.999 INFO:teuthology.orchestra.run.smithi018.stdout: osd mclock profile = high_recovery_ops 2024-09-16T12:20:25.999 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:25.999 INFO:teuthology.orchestra.run.smithi018.stdout: osd recover clone overlap = true 2024-09-16T12:20:25.999 INFO:teuthology.orchestra.run.smithi018.stdout: osd recovery max chunk = 1048576 2024-09-16T12:20:25.999 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:25.999 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug shutdown = true 2024-09-16T12:20:25.999 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug op order = true 2024-09-16T12:20:25.999 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug verify stray on activate = true 2024-09-16T12:20:25.999 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:25.999 INFO:teuthology.orchestra.run.smithi018.stdout: osd open classes on start = true 2024-09-16T12:20:26.000 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug pg log writeout = true 2024-09-16T12:20:26.000 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:26.000 INFO:teuthology.orchestra.run.smithi018.stdout: osd deep scrub update digest min age = 30 2024-09-16T12:20:26.000 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:26.000 INFO:teuthology.orchestra.run.smithi018.stdout: osd map max advance = 10 2024-09-16T12:20:26.000 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:26.000 INFO:teuthology.orchestra.run.smithi018.stdout: journal zero on create = true 2024-09-16T12:20:26.000 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:26.000 INFO:teuthology.orchestra.run.smithi018.stdout: filestore ondisk finisher threads = 3 2024-09-16T12:20:26.000 INFO:teuthology.orchestra.run.smithi018.stdout: filestore apply finisher threads = 3 2024-09-16T12:20:26.000 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:26.000 INFO:teuthology.orchestra.run.smithi018.stdout: bdev debug aio = true 2024-09-16T12:20:26.000 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug misdirected ops = true 2024-09-16T12:20:26.001 INFO:teuthology.orchestra.run.smithi018.stdout: bluefs allocator = hybrid 2024-09-16T12:20:26.001 INFO:teuthology.orchestra.run.smithi018.stdout: bluestore allocator = hybrid 2024-09-16T12:20:26.001 INFO:teuthology.orchestra.run.smithi018.stdout: bluestore block size = 96636764160 2024-09-16T12:20:26.001 INFO:teuthology.orchestra.run.smithi018.stdout: bluestore fsck on mount = True 2024-09-16T12:20:26.001 INFO:teuthology.orchestra.run.smithi018.stdout: debug bluefs = 1/20 2024-09-16T12:20:26.001 INFO:teuthology.orchestra.run.smithi018.stdout: debug bluestore = 1/20 2024-09-16T12:20:26.001 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2024-09-16T12:20:26.001 INFO:teuthology.orchestra.run.smithi018.stdout: debug osd = 20 2024-09-16T12:20:26.001 INFO:teuthology.orchestra.run.smithi018.stdout: debug rocksdb = 4/10 2024-09-16T12:20:26.001 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T12:20:26.001 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd full ratio = 0.9 2024-09-16T12:20:26.001 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd nearfull ratio = 0.8 2024-09-16T12:20:26.001 INFO:teuthology.orchestra.run.smithi018.stdout: osd failsafe full ratio = 0.95 2024-09-16T12:20:26.002 INFO:teuthology.orchestra.run.smithi018.stdout: osd objectstore = bluestore 2024-09-16T12:20:26.002 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:26.002 INFO:teuthology.orchestra.run.smithi018.stdout:[mgr] 2024-09-16T12:20:26.002 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2024-09-16T12:20:26.002 INFO:teuthology.orchestra.run.smithi018.stdout: debug mgr = 20 2024-09-16T12:20:26.002 INFO:teuthology.orchestra.run.smithi018.stdout: debug mon = 20 2024-09-16T12:20:26.002 INFO:teuthology.orchestra.run.smithi018.stdout: debug auth = 20 2024-09-16T12:20:26.002 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min pgs per osd = 4 2024-09-16T12:20:26.002 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min bytes per osd = 10 2024-09-16T12:20:26.002 INFO:teuthology.orchestra.run.smithi018.stdout: mgr/telemetry/nag = false 2024-09-16T12:20:26.002 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:26.002 INFO:teuthology.orchestra.run.smithi018.stdout:[mon] 2024-09-16T12:20:26.002 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2024-09-16T12:20:26.003 INFO:teuthology.orchestra.run.smithi018.stdout: debug mon = 20 2024-09-16T12:20:26.003 INFO:teuthology.orchestra.run.smithi018.stdout: debug paxos = 20 2024-09-16T12:20:26.003 INFO:teuthology.orchestra.run.smithi018.stdout: debug auth = 20 2024-09-16T12:20:26.003 INFO:teuthology.orchestra.run.smithi018.stdout: mon data avail warn = 5 2024-09-16T12:20:26.003 INFO:teuthology.orchestra.run.smithi018.stdout: mon mgr mkfs grace = 240 2024-09-16T12:20:26.003 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min pgs per osd = 4 2024-09-16T12:20:26.003 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd reporter subtree level = osd 2024-09-16T12:20:26.003 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd prime pg temp = true 2024-09-16T12:20:26.003 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min bytes per osd = 10 2024-09-16T12:20:26.003 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:26.003 INFO:teuthology.orchestra.run.smithi018.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T12:20:26.003 INFO:teuthology.orchestra.run.smithi018.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T12:20:26.003 INFO:teuthology.orchestra.run.smithi018.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T12:20:26.004 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:26.004 INFO:teuthology.orchestra.run.smithi018.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T12:20:26.004 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T12:20:26.004 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T12:20:26.004 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:26.004 INFO:teuthology.orchestra.run.smithi018.stdout: # 1m isn't quite enough 2024-09-16T12:20:26.004 INFO:teuthology.orchestra.run.smithi018.stdout: mon_down_mkfs_grace = 2m 2024-09-16T12:20:26.004 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:26.004 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_filestore_osds = false 2024-09-16T12:20:26.004 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:26.004 INFO:teuthology.orchestra.run.smithi018.stdout:[client] 2024-09-16T12:20:26.004 INFO:teuthology.orchestra.run.smithi018.stdout: rgw cache enabled = true 2024-09-16T12:20:26.004 INFO:teuthology.orchestra.run.smithi018.stdout: rgw enable ops log = true 2024-09-16T12:20:26.005 INFO:teuthology.orchestra.run.smithi018.stdout: rgw enable usage log = true 2024-09-16T12:20:26.005 INFO:teuthology.orchestra.run.smithi018.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T12:20:26.005 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T12:20:26.005 INFO:teuthology.orchestra.run.smithi018.stdout: debug rbd = 20 2024-09-16T12:20:26.005 INFO:teuthology.orchestra.run.smithi018.stdout: debug rbd_mirror = 15 2024-09-16T12:20:26.005 INFO:teuthology.orchestra.run.smithi018.stdout: log to stderr = False 2024-09-16T12:20:26.005 INFO:teuthology.orchestra.run.smithi018.stdout: rbd default features = 125 2024-09-16T12:20:26.005 INFO:teuthology.orchestra.run.smithi018.stdout:[mon.a] 2024-09-16T12:20:26.005 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.0] 2024-09-16T12:20:26.005 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:26.005 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:26.005 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.1] 2024-09-16T12:20:26.006 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:26.006 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:26.006 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.2] 2024-09-16T12:20:26.006 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:26.006 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:26.006 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.3] 2024-09-16T12:20:26.006 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:26.006 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:26.006 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.4] 2024-09-16T12:20:26.006 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:26.006 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:26.006 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.5] 2024-09-16T12:20:26.006 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:26.006 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:26.007 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.6] 2024-09-16T12:20:26.007 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:20:26.007 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:20:26.007 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-09-16T12:20:26.007 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-09-16T12:20:26.056 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T12:20:26.057 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:20:26.057 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-09-16T12:20:26.064 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:20:26.064 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-09-16T12:20:26.071 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi018.front.sepia.ceph.com 2024-09-16T12:20:26.071 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:20:26.071 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-16T12:20:26.071 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-16T12:20:26.093 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:20:26.093 DEBUG:teuthology.orchestra.run.smithi018:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-16T12:20:26.099 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2024-09-16T12:20:26.099 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:20:26.099 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-16T12:20:26.099 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-16T12:20:26.127 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:20:26.127 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-16T12:20:26.177 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T12:20:26.177 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T12:20:26.178 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-16T12:20:26.226 INFO:teuthology.orchestra.run.smithi018.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-16T12:20:26.229 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T12:20:26.229 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T12:20:26.229 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-09-16T12:20:26.263 INFO:teuthology.orchestra.run.smithi018.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-09-16T12:20:26.277 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-09-16T12:20:26.313 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-09-16T12:20:26.326 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-16T12:20:26.358 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-16T12:20:26.370 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-16T12:20:26.447 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-16T12:20:26.460 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-16T12:20:26.492 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-16T12:20:26.503 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-16T12:20:26.534 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-16T12:20:26.547 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-16T12:20:26.628 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-16T12:20:26.641 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-16T12:20:26.673 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-16T12:20:26.684 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-16T12:20:26.717 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-16T12:20:26.729 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T12:20:26.729 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi018.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T12:20:26.730 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-09-16T12:20:26.745 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T12:20:26.745 INFO:tasks.ceph:role: cluster1.osd.0 2024-09-16T12:20:26.745 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi018.front.sepia.ceph.com 2024-09-16T12:20:26.746 DEBUG:teuthology.orchestra.run.smithi018:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T12:20:26.788 INFO:teuthology.orchestra.run.smithi018.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T12:20:26.788 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T12:20:26.788 INFO:teuthology.orchestra.run.smithi018.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T12:20:26.788 INFO:teuthology.orchestra.run.smithi018.stdout: = reflink=1 2024-09-16T12:20:26.788 INFO:teuthology.orchestra.run.smithi018.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T12:20:26.788 INFO:teuthology.orchestra.run.smithi018.stdout: = sunit=0 swidth=0 blks 2024-09-16T12:20:26.789 INFO:teuthology.orchestra.run.smithi018.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T12:20:26.789 INFO:teuthology.orchestra.run.smithi018.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T12:20:26.789 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T12:20:26.789 INFO:teuthology.orchestra.run.smithi018.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T12:20:26.790 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi018.front.sepia.ceph.com -o noatime 2024-09-16T12:20:26.790 DEBUG:teuthology.orchestra.run.smithi018:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-09-16T12:20:26.854 DEBUG:teuthology.orchestra.run.smithi018:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-09-16T12:20:26.863 INFO:teuthology.orchestra.run.smithi018.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T12:20:26.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:20:26.864 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-09-16T12:20:26.874 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T12:20:26.874 INFO:tasks.ceph:role: cluster1.osd.1 2024-09-16T12:20:26.874 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi018.front.sepia.ceph.com 2024-09-16T12:20:26.875 DEBUG:teuthology.orchestra.run.smithi018:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T12:20:26.917 INFO:teuthology.orchestra.run.smithi018.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T12:20:26.917 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T12:20:26.917 INFO:teuthology.orchestra.run.smithi018.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T12:20:26.917 INFO:teuthology.orchestra.run.smithi018.stdout: = reflink=1 2024-09-16T12:20:26.917 INFO:teuthology.orchestra.run.smithi018.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T12:20:26.917 INFO:teuthology.orchestra.run.smithi018.stdout: = sunit=0 swidth=0 blks 2024-09-16T12:20:26.917 INFO:teuthology.orchestra.run.smithi018.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T12:20:26.917 INFO:teuthology.orchestra.run.smithi018.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T12:20:26.918 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T12:20:26.918 INFO:teuthology.orchestra.run.smithi018.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T12:20:26.919 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi018.front.sepia.ceph.com -o noatime 2024-09-16T12:20:26.919 DEBUG:teuthology.orchestra.run.smithi018:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-09-16T12:20:26.984 DEBUG:teuthology.orchestra.run.smithi018:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-09-16T12:20:26.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:20:26.996 INFO:teuthology.orchestra.run.smithi018.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T12:20:26.996 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-09-16T12:20:27.006 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T12:20:27.006 INFO:tasks.ceph:role: cluster1.osd.2 2024-09-16T12:20:27.006 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi018.front.sepia.ceph.com 2024-09-16T12:20:27.007 DEBUG:teuthology.orchestra.run.smithi018:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T12:20:27.046 INFO:teuthology.orchestra.run.smithi018.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T12:20:27.046 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T12:20:27.046 INFO:teuthology.orchestra.run.smithi018.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T12:20:27.046 INFO:teuthology.orchestra.run.smithi018.stdout: = reflink=1 2024-09-16T12:20:27.046 INFO:teuthology.orchestra.run.smithi018.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T12:20:27.046 INFO:teuthology.orchestra.run.smithi018.stdout: = sunit=0 swidth=0 blks 2024-09-16T12:20:27.047 INFO:teuthology.orchestra.run.smithi018.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T12:20:27.047 INFO:teuthology.orchestra.run.smithi018.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T12:20:27.047 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T12:20:27.047 INFO:teuthology.orchestra.run.smithi018.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T12:20:27.048 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi018.front.sepia.ceph.com -o noatime 2024-09-16T12:20:27.048 DEBUG:teuthology.orchestra.run.smithi018:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-09-16T12:20:27.069 DEBUG:teuthology.orchestra.run.smithi018:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-09-16T12:20:27.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:20:27.081 INFO:teuthology.orchestra.run.smithi018.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T12:20:27.081 DEBUG:teuthology.orchestra.run.smithi018:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T12:20:27.201 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:27.191+0000 7ff82cea3540 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-09-16T12:20:27.201 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:27.191+0000 7ff82cea3540 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-09-16T12:20:27.201 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:27.191+0000 7ff82cea3540 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-09-16T12:20:27.201 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:27.191+0000 7ff82cea3540 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-09-16T12:20:27.201 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:27.191+0000 7ff82cea3540 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2024-09-16T12:20:27.202 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:27.191+0000 7ff82cea3540 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-09-16T12:20:27.755 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:27.747+0000 7ff82cea3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:20:28.534 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:28.527+0000 7ff82cea3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:20:29.316 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:29.307+0000 7ff82cea3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:20:30.182 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-09-16T12:20:30.198 DEBUG:teuthology.orchestra.run.smithi018:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T12:20:30.239 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:30.231+0000 7f5c50ffa540 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-09-16T12:20:30.239 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:30.231+0000 7f5c50ffa540 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-09-16T12:20:30.239 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:30.231+0000 7f5c50ffa540 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-09-16T12:20:30.239 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:30.231+0000 7f5c50ffa540 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-09-16T12:20:30.239 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:30.231+0000 7f5c50ffa540 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2024-09-16T12:20:30.239 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:30.231+0000 7f5c50ffa540 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-09-16T12:20:30.791 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:30.783+0000 7f5c50ffa540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:20:31.576 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:31.567+0000 7f5c50ffa540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:20:32.360 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:32.351+0000 7f5c50ffa540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:20:33.218 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-09-16T12:20:33.233 DEBUG:teuthology.orchestra.run.smithi018:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T12:20:33.316 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:33.307+0000 7f1b5d9c2540 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-09-16T12:20:33.317 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:33.307+0000 7f1b5d9c2540 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-09-16T12:20:33.317 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:33.307+0000 7f1b5d9c2540 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-09-16T12:20:33.317 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:33.307+0000 7f1b5d9c2540 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-09-16T12:20:33.317 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:33.307+0000 7f1b5d9c2540 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2024-09-16T12:20:33.317 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:33.307+0000 7f1b5d9c2540 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-09-16T12:20:33.869 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:33.859+0000 7f1b5d9c2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:20:34.647 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:34.639+0000 7f1b5d9c2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:20:35.429 INFO:teuthology.orchestra.run.smithi018.stderr:2024-09-16T12:20:35.419+0000 7f1b5d9c2540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:20:36.281 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-09-16T12:20:36.298 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T12:20:36.298 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:20:36.298 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-09-16T12:20:36.309 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:20:36.309 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-09-16T12:20:36.365 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:20:36.365 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-09-16T12:20:36.377 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:20:36.377 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-09-16T12:20:36.388 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:20:36.388 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-09-16T12:20:36.436 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:20:36.436 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-09-16T12:20:36.444 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:20:36.444 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-09-16T12:20:36.493 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:20:36.493 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-09-16T12:20:36.500 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:20:36.500 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-09-16T12:20:36.507 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:20:36.507 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-09-16T12:20:36.513 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:20:36.513 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-09-16T12:20:36.562 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:20:36.562 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-09-16T12:20:36.568 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:20:36.568 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-09-16T12:20:36.574 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T12:20:36.574 DEBUG:teuthology.orchestra.run.smithi018:> sudo tee -a /etc/ceph/cluster1.keyring 2024-09-16T12:20:36.589 INFO:teuthology.orchestra.run.smithi018.stdout:[mgr.x] 2024-09-16T12:20:36.589 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCKIuhmdmlPDRAAoXcdY1ZnoIwcQBUDGEmqMw== 2024-09-16T12:20:36.589 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.0] 2024-09-16T12:20:36.589 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCLIuhmWiTCCxAAhNZvA4mBOr+bKIypYOJSeQ== 2024-09-16T12:20:36.589 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.1] 2024-09-16T12:20:36.589 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCOIuhmuroCDhAAClI3Ruy6o+0HB0iMijLX8g== 2024-09-16T12:20:36.589 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.2] 2024-09-16T12:20:36.589 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCRIuhmiQCnEhAA215f2bCsgZdKbfRew72nEA== 2024-09-16T12:20:36.589 INFO:teuthology.orchestra.run.smithi018.stdout:[client.0] 2024-09-16T12:20:36.589 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCKIuhmK+KCDxAAK55nolsvTKnZBgNeYO+c9A== 2024-09-16T12:20:36.590 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror] 2024-09-16T12:20:36.590 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCKIuhmFCRqEhAABUxSoCEvUpWAWagbf9vCnQ== 2024-09-16T12:20:36.590 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.0] 2024-09-16T12:20:36.590 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCKIuhmENYjFRAAXIhRGdrgq+71DkKBG5+EYA== 2024-09-16T12:20:36.590 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.1] 2024-09-16T12:20:36.590 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCKIuhm5RF1GhAAJ/+RWgzCuwCZabW0QiRRlw== 2024-09-16T12:20:36.590 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.2] 2024-09-16T12:20:36.590 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCKIuhm02YfHRAAo7SzHfqzsOArtzCrBu0I6A== 2024-09-16T12:20:36.590 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.3] 2024-09-16T12:20:36.590 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCKIuhmhpuhHxAA+ZXrPUA+11gFS9YA8VROmQ== 2024-09-16T12:20:36.590 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.4] 2024-09-16T12:20:36.590 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCKIuhmngoyJRAAaJk6Sl3fFG6DUCKU3hWmew== 2024-09-16T12:20:36.591 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.5] 2024-09-16T12:20:36.591 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCKIuhm7Q/lJxAA/IMnRXGG6EGP2o49UWJlCA== 2024-09-16T12:20:36.591 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.6] 2024-09-16T12:20:36.591 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQCKIuhm4AOMKhAA+7n4FN4VaMfCkcrrXd5KWw== 2024-09-16T12:20:36.592 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T12:20:36.633 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T12:20:36.718 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T12:20:36.757 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T12:20:36.797 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:20:36.888 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:20:36.936 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:20:36.984 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:20:37.026 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:20:37.071 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:20:37.112 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:20:37.200 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:20:37.246 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:20:37.290 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T12:20:37.290 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-09-16T12:20:37.303 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-09-16T12:20:37.400 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-09-16T12:20:37.412 DEBUG:teuthology.orchestra.run.smithi018:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-09-16T12:20:37.417 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-09-16T12:20:37.417 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-09-16T12:20:37.418 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-09-16T12:20:37.420 INFO:tasks.ceph.cluster1.mon.a:Started 2024-09-16T12:20:37.420 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-09-16T12:20:37.420 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-09-16T12:20:37.421 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-09-16T12:20:37.423 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-09-16T12:20:37.423 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T12:20:37.423 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph --cluster cluster1 osd crush tunables default 2024-09-16T12:20:37.630 INFO:teuthology.orchestra.run.smithi018.stderr:adjusted tunables profile to default 2024-09-16T12:20:37.641 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-09-16T12:20:37.641 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:20:37.641 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-09-16T12:20:37.655 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:20:37.655 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-09-16T12:20:37.667 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:20:37.667 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-09-16T12:20:37.679 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph --cluster cluster1 osd new 2f63d01a-c8dc-4d07-bb9e-7866d5784dd6 0 2024-09-16T12:20:37.724 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:37.715+0000 7fddaf5cc000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T12:20:37.886 INFO:teuthology.orchestra.run.smithi018.stdout:0 2024-09-16T12:20:37.898 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph --cluster cluster1 osd new dd97ec5a-4f4a-4a34-82c1-403d5b9ac50b 1 2024-09-16T12:20:38.093 INFO:teuthology.orchestra.run.smithi018.stdout:1 2024-09-16T12:20:38.105 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph --cluster cluster1 osd new 0f4108f3-b884-44c1-b94e-e3a5f8f6d4d3 2 2024-09-16T12:20:38.307 INFO:teuthology.orchestra.run.smithi018.stdout:2 2024-09-16T12:20:38.323 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-09-16T12:20:38.323 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-09-16T12:20:38.325 INFO:tasks.ceph.cluster1.osd.0:Started 2024-09-16T12:20:38.325 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-09-16T12:20:38.325 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-09-16T12:20:38.327 INFO:tasks.ceph.cluster1.osd.1:Started 2024-09-16T12:20:38.327 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-09-16T12:20:38.327 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-09-16T12:20:38.329 INFO:tasks.ceph.cluster1.osd.2:Started 2024-09-16T12:20:38.329 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T12:20:38.460 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:38.451+0000 7fddaf5cc000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T12:20:38.519 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:38.511+0000 7fddaf5cc000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T12:20:38.638 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:38.627+0000 7fddaf5cc000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T12:20:38.704 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:20:38.695+0000 7ff5071c4540 -1 Falling back to public interface 2024-09-16T12:20:38.709 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:38.709 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":5,"fsid":"3d80de80-eeb4-495e-a052-ec9a35b35bbd","created":"2024-09-16T12:20:37.527152+0000","modified":"2024-09-16T12:20:38.300635+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T12:20:38.300635+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"2f63d01a-c8dc-4d07-bb9e-7866d5784dd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dd97ec5a-4f4a-4a34-82c1-403d5b9ac50b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0f4108f3-b884-44c1-b94e-e3a5f8f6d4d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:20:38.719 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:20:38.711+0000 7f61c20d3540 -1 Falling back to public interface 2024-09-16T12:20:38.723 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-09-16T12:20:38.723 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T12:20:38.882 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:38.875+0000 7fddaf5cc000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T12:20:38.936 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:38.927+0000 7fddaf5cc000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T12:20:38.972 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:20:38.963+0000 7f40824ad540 -1 Falling back to public interface 2024-09-16T12:20:39.025 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-16T12:20:39.130 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:39.119+0000 7fddaf5cc000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T12:20:39.195 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:39.187+0000 7fddaf5cc000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T12:20:39.201 INFO:teuthology.misc.health.smithi018.stdout: 2024-09-16T12:20:39.201 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":5,"fsid":"3d80de80-eeb4-495e-a052-ec9a35b35bbd","created":"2024-09-16T12:20:37.527152+0000","modified":"2024-09-16T12:20:38.300635+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T12:20:38.300635+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"2f63d01a-c8dc-4d07-bb9e-7866d5784dd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dd97ec5a-4f4a-4a34-82c1-403d5b9ac50b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0f4108f3-b884-44c1-b94e-e3a5f8f6d4d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:20:39.217 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T12:20:39.262 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:39.251+0000 7fddaf5cc000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T12:20:39.695 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:39.687+0000 7fddaf5cc000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T12:20:39.820 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:39.811+0000 7fddaf5cc000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T12:20:40.147 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:40.139+0000 7fddaf5cc000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T12:20:40.325 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:40.315+0000 7fddaf5cc000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T12:20:40.390 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:40.379+0000 7fddaf5cc000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T12:20:40.445 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:40.435+0000 7fddaf5cc000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T12:20:40.505 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:40.495+0000 7fddaf5cc000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T12:20:40.680 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:40.671+0000 7fddaf5cc000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T12:20:40.847 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:40.839+0000 7fddaf5cc000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T12:20:40.916 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:40.907+0000 7fddaf5cc000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T12:20:41.078 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:41.067+0000 7fddaf5cc000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T12:20:41.149 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:41.139+0000 7fddaf5cc000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T12:20:41.905 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:41.895+0000 7fddaf5cc000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T12:20:42.421 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:20:42.411+0000 7ff5071c4540 -1 osd.0 0 log_to_monitors true 2024-09-16T12:20:42.618 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:20:42.607+0000 7f61c20d3540 -1 osd.2 0 log_to_monitors true 2024-09-16T12:20:42.809 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:20:42.799+0000 7f40824ad540 -1 osd.1 0 log_to_monitors true 2024-09-16T12:20:42.937 INFO:tasks.ceph.cluster1.mgr.x.smithi018.stderr:2024-09-16T12:20:42.927+0000 7fdd98641700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-16T12:20:45.492 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:20:45.482+0000 7f407dc4d700 -1 osd.1 0 waiting for initial osdmap 2024-09-16T12:20:45.500 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:20:45.490+0000 7f40769f0700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T12:20:45.520 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-16T12:20:45.520 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:20:45.510+0000 7f61bd873700 -1 osd.2 0 waiting for initial osdmap 2024-09-16T12:20:45.526 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:20:45.518+0000 7f61b6616700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T12:20:45.552 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:20:45.542+0000 7ff502964700 -1 osd.0 0 waiting for initial osdmap 2024-09-16T12:20:45.559 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:20:45.550+0000 7ff4fb707700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T12:20:45.778 INFO:teuthology.misc.health.smithi018.stdout: 2024-09-16T12:20:45.778 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":7,"fsid":"3d80de80-eeb4-495e-a052-ec9a35b35bbd","created":"2024-09-16T12:20:37.527152+0000","modified":"2024-09-16T12:20:43.982260+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T12:20:38.300635+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"2f63d01a-c8dc-4d07-bb9e-7866d5784dd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"dd97ec5a-4f4a-4a34-82c1-403d5b9ac50b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0f4108f3-b884-44c1-b94e-e3a5f8f6d4d3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:20:45.789 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T12:20:52.093 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-16T12:20:52.386 INFO:teuthology.misc.health.smithi018.stdout: 2024-09-16T12:20:52.386 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":12,"fsid":"3d80de80-eeb4-495e-a052-ec9a35b35bbd","created":"2024-09-16T12:20:37.527152+0000","modified":"2024-09-16T12:20:50.531883+0000","last_up_change":"2024-09-16T12:20:46.517261+0000","last_in_change":"2024-09-16T12:20:38.300635+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T12:20:47.990458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"2f63d01a-c8dc-4d07-bb9e-7866d5784dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":21164},{"type":"v1","addr":"172.21.15.18:6801","nonce":21164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":21164},{"type":"v1","addr":"172.21.15.18:6803","nonce":21164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":21164},{"type":"v1","addr":"172.21.15.18:6807","nonce":21164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":21164},{"type":"v1","addr":"172.21.15.18:6805","nonce":21164}]},"public_addr":"172.21.15.18:6801/21164","cluster_addr":"172.21.15.18:6803/21164","heartbeat_back_addr":"172.21.15.18:6807/21164","heartbeat_front_addr":"172.21.15.18:6805/21164","state":["exists","up"]},{"osd":1,"uuid":"dd97ec5a-4f4a-4a34-82c1-403d5b9ac50b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6816","nonce":21167},{"type":"v1","addr":"172.21.15.18:6817","nonce":21167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6818","nonce":21167},{"type":"v1","addr":"172.21.15.18:6819","nonce":21167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6822","nonce":21167},{"type":"v1","addr":"172.21.15.18:6823","nonce":21167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6820","nonce":21167},{"type":"v1","addr":"172.21.15.18:6821","nonce":21167}]},"public_addr":"172.21.15.18:6817/21167","cluster_addr":"172.21.15.18:6819/21167","heartbeat_back_addr":"172.21.15.18:6823/21167","heartbeat_front_addr":"172.21.15.18:6821/21167","state":["exists","up"]},{"osd":2,"uuid":"0f4108f3-b884-44c1-b94e-e3a5f8f6d4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6808","nonce":21165},{"type":"v1","addr":"172.21.15.18:6809","nonce":21165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6810","nonce":21165},{"type":"v1","addr":"172.21.15.18:6811","nonce":21165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6814","nonce":21165},{"type":"v1","addr":"172.21.15.18:6815","nonce":21165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6812","nonce":21165},{"type":"v1","addr":"172.21.15.18:6813","nonce":21165}]},"public_addr":"172.21.15.18:6809/21165","cluster_addr":"172.21.15.18:6811/21165","heartbeat_back_addr":"172.21.15.18:6815/21165","heartbeat_front_addr":"172.21.15.18:6813/21165","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:20:43.407329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:20:43.768607+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:20:43.654909+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:20:52.398 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-16T12:20:52.398 INFO:tasks.ceph:Creating RBD pool 2024-09-16T12:20:52.399 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-09-16T12:20:53.561 INFO:teuthology.orchestra.run.smithi018.stderr:pool 'rbd' created 2024-09-16T12:20:53.573 DEBUG:teuthology.orchestra.run.smithi018:> rbd --cluster cluster1 pool init rbd 2024-09-16T12:20:56.583 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-09-16T12:20:56.584 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-09-16T12:20:56.584 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-09-16T12:20:56.584 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-09-16T12:20:56.585 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T12:20:56.926 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:56.941 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":5,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6824","nonce":20981},{"type":"v1","addr":"172.21.15.18:6825","nonce":20981}]},"active_addr":"172.21.15.18:6825/20981","active_change":"2024-09-16T12:20:41.920623+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.18:0","nonce":3448788695}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.18:0","nonce":3276538571}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.18:0","nonce":3265004615}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.18:0","nonce":431606287}]}]} 2024-09-16T12:20:56.943 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-09-16T12:20:56.943 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-09-16T12:20:56.943 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T12:20:57.231 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:57.231 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":16,"fsid":"3d80de80-eeb4-495e-a052-ec9a35b35bbd","created":"2024-09-16T12:20:37.527152+0000","modified":"2024-09-16T12:20:56.559723+0000","last_up_change":"2024-09-16T12:20:46.517261+0000","last_in_change":"2024-09-16T12:20:38.300635+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T12:20:47.990458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T12:20:52.731352+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2f63d01a-c8dc-4d07-bb9e-7866d5784dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":21164},{"type":"v1","addr":"172.21.15.18:6801","nonce":21164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":21164},{"type":"v1","addr":"172.21.15.18:6803","nonce":21164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":21164},{"type":"v1","addr":"172.21.15.18:6807","nonce":21164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":21164},{"type":"v1","addr":"172.21.15.18:6805","nonce":21164}]},"public_addr":"172.21.15.18:6801/21164","cluster_addr":"172.21.15.18:6803/21164","heartbeat_back_addr":"172.21.15.18:6807/21164","heartbeat_front_addr":"172.21.15.18:6805/21164","state":["exists","up"]},{"osd":1,"uuid":"dd97ec5a-4f4a-4a34-82c1-403d5b9ac50b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6816","nonce":21167},{"type":"v1","addr":"172.21.15.18:6817","nonce":21167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6818","nonce":21167},{"type":"v1","addr":"172.21.15.18:6819","nonce":21167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6822","nonce":21167},{"type":"v1","addr":"172.21.15.18:6823","nonce":21167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6820","nonce":21167},{"type":"v1","addr":"172.21.15.18:6821","nonce":21167}]},"public_addr":"172.21.15.18:6817/21167","cluster_addr":"172.21.15.18:6819/21167","heartbeat_back_addr":"172.21.15.18:6823/21167","heartbeat_front_addr":"172.21.15.18:6821/21167","state":["exists","up"]},{"osd":2,"uuid":"0f4108f3-b884-44c1-b94e-e3a5f8f6d4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6808","nonce":21165},{"type":"v1","addr":"172.21.15.18:6809","nonce":21165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6810","nonce":21165},{"type":"v1","addr":"172.21.15.18:6811","nonce":21165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6814","nonce":21165},{"type":"v1","addr":"172.21.15.18:6815","nonce":21165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6812","nonce":21165},{"type":"v1","addr":"172.21.15.18:6813","nonce":21165}]},"public_addr":"172.21.15.18:6809/21165","cluster_addr":"172.21.15.18:6811/21165","heartbeat_back_addr":"172.21.15.18:6815/21165","heartbeat_front_addr":"172.21.15.18:6813/21165","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:20:43.407329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:20:43.768607+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:20:43.654909+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:20:57.245 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-09-16T12:20:57.246 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T12:20:57.540 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:57.541 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":16,"fsid":"3d80de80-eeb4-495e-a052-ec9a35b35bbd","created":"2024-09-16T12:20:37.527152+0000","modified":"2024-09-16T12:20:56.559723+0000","last_up_change":"2024-09-16T12:20:46.517261+0000","last_in_change":"2024-09-16T12:20:38.300635+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T12:20:47.990458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T12:20:52.731352+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2f63d01a-c8dc-4d07-bb9e-7866d5784dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":21164},{"type":"v1","addr":"172.21.15.18:6801","nonce":21164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":21164},{"type":"v1","addr":"172.21.15.18:6803","nonce":21164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":21164},{"type":"v1","addr":"172.21.15.18:6807","nonce":21164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":21164},{"type":"v1","addr":"172.21.15.18:6805","nonce":21164}]},"public_addr":"172.21.15.18:6801/21164","cluster_addr":"172.21.15.18:6803/21164","heartbeat_back_addr":"172.21.15.18:6807/21164","heartbeat_front_addr":"172.21.15.18:6805/21164","state":["exists","up"]},{"osd":1,"uuid":"dd97ec5a-4f4a-4a34-82c1-403d5b9ac50b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6816","nonce":21167},{"type":"v1","addr":"172.21.15.18:6817","nonce":21167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6818","nonce":21167},{"type":"v1","addr":"172.21.15.18:6819","nonce":21167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6822","nonce":21167},{"type":"v1","addr":"172.21.15.18:6823","nonce":21167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6820","nonce":21167},{"type":"v1","addr":"172.21.15.18:6821","nonce":21167}]},"public_addr":"172.21.15.18:6817/21167","cluster_addr":"172.21.15.18:6819/21167","heartbeat_back_addr":"172.21.15.18:6823/21167","heartbeat_front_addr":"172.21.15.18:6821/21167","state":["exists","up"]},{"osd":2,"uuid":"0f4108f3-b884-44c1-b94e-e3a5f8f6d4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6808","nonce":21165},{"type":"v1","addr":"172.21.15.18:6809","nonce":21165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6810","nonce":21165},{"type":"v1","addr":"172.21.15.18:6811","nonce":21165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6814","nonce":21165},{"type":"v1","addr":"172.21.15.18:6815","nonce":21165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6812","nonce":21165},{"type":"v1","addr":"172.21.15.18:6813","nonce":21165}]},"public_addr":"172.21.15.18:6809/21165","cluster_addr":"172.21.15.18:6811/21165","heartbeat_back_addr":"172.21.15.18:6815/21165","heartbeat_front_addr":"172.21.15.18:6813/21165","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:20:43.407329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:20:43.768607+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:20:43.654909+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:20:57.552 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-09-16T12:20:57.552 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-09-16T12:20:57.552 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-09-16T12:20:57.686 INFO:teuthology.orchestra.run.smithi018.stdout:34359738374 2024-09-16T12:20:57.686 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-16T12:20:57.696 INFO:teuthology.orchestra.run.smithi018.stdout:34359738375 2024-09-16T12:20:57.696 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-16T12:20:57.708 INFO:teuthology.orchestra.run.smithi018.stdout:34359738374 2024-09-16T12:20:57.709 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-16T12:20:57.971 INFO:teuthology.orchestra.run.smithi018.stdout:34359738373 2024-09-16T12:20:57.983 INFO:teuthology.orchestra.run.smithi018.stdout:34359738372 2024-09-16T12:20:57.984 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738375 got 34359738373 for osd.0 2024-09-16T12:20:57.995 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738372 for osd.1 2024-09-16T12:20:58.036 INFO:teuthology.orchestra.run.smithi018.stdout:34359738372 2024-09-16T12:20:58.050 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738372 for osd.2 2024-09-16T12:20:58.984 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-16T12:20:58.997 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-16T12:20:59.051 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-16T12:20:59.299 INFO:teuthology.orchestra.run.smithi018.stdout:34359738375 2024-09-16T12:20:59.312 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738375 got 34359738375 for osd.0 2024-09-16T12:20:59.313 DEBUG:teuthology.parallel:result is None 2024-09-16T12:20:59.334 INFO:teuthology.orchestra.run.smithi018.stdout:34359738374 2024-09-16T12:20:59.334 INFO:teuthology.orchestra.run.smithi018.stdout:34359738374 2024-09-16T12:20:59.348 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.1 2024-09-16T12:20:59.348 DEBUG:teuthology.parallel:result is None 2024-09-16T12:20:59.349 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738374 got 34359738374 for osd.2 2024-09-16T12:20:59.349 DEBUG:teuthology.parallel:result is None 2024-09-16T12:20:59.349 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-16T12:20:59.349 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T12:20:59.639 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:59.643 INFO:teuthology.orchestra.run.smithi018.stderr:dumped all 2024-09-16T12:20:59.656 INFO:teuthology.orchestra.run.smithi018.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-16T12:20:57.931731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":910308,"kb_used_data":1576,"kb_used_omap":4,"kb_used_meta":908667,"kb_avail":282205212,"statfs":{"total":289910292480,"available":288978137088,"internally_reserved":0,"allocated":1613824,"data_stored":1302466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":930475016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.387845"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:20:56.573780+0000","last_change":"2024-09-16T12:20:56.574082+0000","last_active":"2024-09-16T12:20:56.573780+0000","last_peered":"2024-09-16T12:20:56.573780+0000","last_clean":"2024-09-16T12:20:56.573780+0000","last_became_active":"2024-09-16T12:20:54.569007+0000","last_became_peered":"2024-09-16T12:20:54.569007+0000","last_unstale":"2024-09-16T12:20:56.573780+0000","last_undegraded":"2024-09-16T12:20:56.573780+0000","last_fullsized":"2024-09-16T12:20:56.573780+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:44:06.351794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055482299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:20:56.573887+0000","last_change":"2024-09-16T12:20:56.574146+0000","last_active":"2024-09-16T12:20:56.573887+0000","last_peered":"2024-09-16T12:20:56.573887+0000","last_clean":"2024-09-16T12:20:56.573887+0000","last_became_active":"2024-09-16T12:20:54.571536+0000","last_became_peered":"2024-09-16T12:20:54.571536+0000","last_unstale":"2024-09-16T12:20:56.573887+0000","last_undegraded":"2024-09-16T12:20:56.573887+0000","last_fullsized":"2024-09-16T12:20:56.573887+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:27.211269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062020100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:20:56.574635+0000","last_change":"2024-09-16T12:20:56.575084+0000","last_active":"2024-09-16T12:20:56.574635+0000","last_peered":"2024-09-16T12:20:56.574635+0000","last_clean":"2024-09-16T12:20:56.574635+0000","last_became_active":"2024-09-16T12:20:54.566716+0000","last_became_peered":"2024-09-16T12:20:54.566716+0000","last_unstale":"2024-09-16T12:20:56.574635+0000","last_undegraded":"2024-09-16T12:20:56.574635+0000","last_fullsized":"2024-09-16T12:20:56.574635+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:07.884926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015293030000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:20:56.574230+0000","last_change":"2024-09-16T12:20:56.574539+0000","last_active":"2024-09-16T12:20:56.574230+0000","last_peered":"2024-09-16T12:20:56.574230+0000","last_clean":"2024-09-16T12:20:56.574230+0000","last_became_active":"2024-09-16T12:20:54.570346+0000","last_became_peered":"2024-09-16T12:20:54.570346+0000","last_unstale":"2024-09-16T12:20:56.574230+0000","last_undegraded":"2024-09-16T12:20:56.574230+0000","last_fullsized":"2024-09-16T12:20:56.574230+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:09:55.562224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086539699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:20:56.568734+0000","last_change":"2024-09-16T12:20:56.568734+0000","last_active":"2024-09-16T12:20:56.568734+0000","last_peered":"2024-09-16T12:20:56.568734+0000","last_clean":"2024-09-16T12:20:56.568734+0000","last_became_active":"2024-09-16T12:20:54.567480+0000","last_became_peered":"2024-09-16T12:20:54.567480+0000","last_unstale":"2024-09-16T12:20:56.568734+0000","last_undegraded":"2024-09-16T12:20:56.568734+0000","last_fullsized":"2024-09-16T12:20:56.568734+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:06.038597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089212600000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:20:56.646836+0000","last_change":"2024-09-16T12:20:56.647363+0000","last_active":"2024-09-16T12:20:56.646836+0000","last_peered":"2024-09-16T12:20:56.646836+0000","last_clean":"2024-09-16T12:20:56.646836+0000","last_became_active":"2024-09-16T12:20:54.564812+0000","last_became_peered":"2024-09-16T12:20:54.564812+0000","last_unstale":"2024-09-16T12:20:56.646836+0000","last_undegraded":"2024-09-16T12:20:56.646836+0000","last_fullsized":"2024-09-16T12:20:56.646836+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:19:30.492388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093612199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:20:56.646685+0000","last_change":"2024-09-16T12:20:56.647314+0000","last_active":"2024-09-16T12:20:56.646685+0000","last_peered":"2024-09-16T12:20:56.646685+0000","last_clean":"2024-09-16T12:20:56.646685+0000","last_became_active":"2024-09-16T12:20:54.566954+0000","last_became_peered":"2024-09-16T12:20:54.566954+0000","last_unstale":"2024-09-16T12:20:56.646685+0000","last_undegraded":"2024-09-16T12:20:56.646685+0000","last_fullsized":"2024-09-16T12:20:56.646685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:04:51.826803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086513400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:20:56.574388+0000","last_change":"2024-09-16T12:20:56.574956+0000","last_active":"2024-09-16T12:20:56.574388+0000","last_peered":"2024-09-16T12:20:56.574388+0000","last_clean":"2024-09-16T12:20:56.574388+0000","last_became_active":"2024-09-16T12:20:54.564938+0000","last_became_peered":"2024-09-16T12:20:54.564938+0000","last_unstale":"2024-09-16T12:20:56.574388+0000","last_undegraded":"2024-09-16T12:20:56.574388+0000","last_fullsized":"2024-09-16T12:20:56.574388+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:03:31.051045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011678050000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'52","reported_seq":80,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:20:55.569528+0000","last_change":"2024-09-16T12:20:50.384662+0000","last_active":"2024-09-16T12:20:55.569528+0000","last_peered":"2024-09-16T12:20:55.569528+0000","last_clean":"2024-09-16T12:20:55.569528+0000","last_became_active":"2024-09-16T12:20:50.381536+0000","last_became_peered":"2024-09-16T12:20:50.381536+0000","last_unstale":"2024-09-16T12:20:55.569528+0000","last_undegraded":"2024-09-16T12:20:55.569528+0000","last_fullsized":"2024-09-16T12:20:55.569528+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:48.519944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:48.519944+0000","last_clean_scrub_stamp":"2024-09-16T12:20:48.519944+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:18.620803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301660,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070180,"statfs":{"total":96636764160,"available":96327864320,"internally_reserved":0,"allocated":139264,"data_stored":40564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306404,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065436,"statfs":{"total":96636764160,"available":96323006464,"internally_reserved":0,"allocated":737280,"data_stored":630951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302244,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069596,"statfs":{"total":96636764160,"available":96327266304,"internally_reserved":0,"allocated":737280,"data_stored":630951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T12:20:59.657 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T12:20:59.947 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:20:59.951 INFO:teuthology.orchestra.run.smithi018.stderr:dumped all 2024-09-16T12:20:59.962 INFO:teuthology.orchestra.run.smithi018.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-16T12:20:59.932177+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":910308,"kb_used_data":1576,"kb_used_omap":4,"kb_used_meta":908667,"kb_avail":282205212,"statfs":{"total":289910292480,"available":288978137088,"internally_reserved":0,"allocated":1613824,"data_stored":1302466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":930475016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001529"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:20:56.573780+0000","last_change":"2024-09-16T12:20:56.574082+0000","last_active":"2024-09-16T12:20:56.573780+0000","last_peered":"2024-09-16T12:20:56.573780+0000","last_clean":"2024-09-16T12:20:56.573780+0000","last_became_active":"2024-09-16T12:20:54.569007+0000","last_became_peered":"2024-09-16T12:20:54.569007+0000","last_unstale":"2024-09-16T12:20:56.573780+0000","last_undegraded":"2024-09-16T12:20:56.573780+0000","last_fullsized":"2024-09-16T12:20:56.573780+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:44:06.351794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055482299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:20:56.573887+0000","last_change":"2024-09-16T12:20:56.574146+0000","last_active":"2024-09-16T12:20:56.573887+0000","last_peered":"2024-09-16T12:20:56.573887+0000","last_clean":"2024-09-16T12:20:56.573887+0000","last_became_active":"2024-09-16T12:20:54.571536+0000","last_became_peered":"2024-09-16T12:20:54.571536+0000","last_unstale":"2024-09-16T12:20:56.573887+0000","last_undegraded":"2024-09-16T12:20:56.573887+0000","last_fullsized":"2024-09-16T12:20:56.573887+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:27.211269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062020100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:20:56.574635+0000","last_change":"2024-09-16T12:20:56.575084+0000","last_active":"2024-09-16T12:20:56.574635+0000","last_peered":"2024-09-16T12:20:56.574635+0000","last_clean":"2024-09-16T12:20:56.574635+0000","last_became_active":"2024-09-16T12:20:54.566716+0000","last_became_peered":"2024-09-16T12:20:54.566716+0000","last_unstale":"2024-09-16T12:20:56.574635+0000","last_undegraded":"2024-09-16T12:20:56.574635+0000","last_fullsized":"2024-09-16T12:20:56.574635+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:07.884926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015293030000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:20:56.574230+0000","last_change":"2024-09-16T12:20:56.574539+0000","last_active":"2024-09-16T12:20:56.574230+0000","last_peered":"2024-09-16T12:20:56.574230+0000","last_clean":"2024-09-16T12:20:56.574230+0000","last_became_active":"2024-09-16T12:20:54.570346+0000","last_became_peered":"2024-09-16T12:20:54.570346+0000","last_unstale":"2024-09-16T12:20:56.574230+0000","last_undegraded":"2024-09-16T12:20:56.574230+0000","last_fullsized":"2024-09-16T12:20:56.574230+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:09:55.562224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086539699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:20:56.568734+0000","last_change":"2024-09-16T12:20:56.568734+0000","last_active":"2024-09-16T12:20:56.568734+0000","last_peered":"2024-09-16T12:20:56.568734+0000","last_clean":"2024-09-16T12:20:56.568734+0000","last_became_active":"2024-09-16T12:20:54.567480+0000","last_became_peered":"2024-09-16T12:20:54.567480+0000","last_unstale":"2024-09-16T12:20:56.568734+0000","last_undegraded":"2024-09-16T12:20:56.568734+0000","last_fullsized":"2024-09-16T12:20:56.568734+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:06.038597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089212600000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:20:56.646836+0000","last_change":"2024-09-16T12:20:56.647363+0000","last_active":"2024-09-16T12:20:56.646836+0000","last_peered":"2024-09-16T12:20:56.646836+0000","last_clean":"2024-09-16T12:20:56.646836+0000","last_became_active":"2024-09-16T12:20:54.564812+0000","last_became_peered":"2024-09-16T12:20:54.564812+0000","last_unstale":"2024-09-16T12:20:56.646836+0000","last_undegraded":"2024-09-16T12:20:56.646836+0000","last_fullsized":"2024-09-16T12:20:56.646836+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:19:30.492388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093612199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:20:56.646685+0000","last_change":"2024-09-16T12:20:56.647314+0000","last_active":"2024-09-16T12:20:56.646685+0000","last_peered":"2024-09-16T12:20:56.646685+0000","last_clean":"2024-09-16T12:20:56.646685+0000","last_became_active":"2024-09-16T12:20:54.566954+0000","last_became_peered":"2024-09-16T12:20:54.566954+0000","last_unstale":"2024-09-16T12:20:56.646685+0000","last_undegraded":"2024-09-16T12:20:56.646685+0000","last_fullsized":"2024-09-16T12:20:56.646685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:04:51.826803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086513400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:20:56.574388+0000","last_change":"2024-09-16T12:20:56.574956+0000","last_active":"2024-09-16T12:20:56.574388+0000","last_peered":"2024-09-16T12:20:56.574388+0000","last_clean":"2024-09-16T12:20:56.574388+0000","last_became_active":"2024-09-16T12:20:54.564938+0000","last_became_peered":"2024-09-16T12:20:54.564938+0000","last_unstale":"2024-09-16T12:20:56.574388+0000","last_undegraded":"2024-09-16T12:20:56.574388+0000","last_fullsized":"2024-09-16T12:20:56.574388+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:03:31.051045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011678050000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"12'52","reported_seq":80,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:20:55.569528+0000","last_change":"2024-09-16T12:20:50.384662+0000","last_active":"2024-09-16T12:20:55.569528+0000","last_peered":"2024-09-16T12:20:55.569528+0000","last_clean":"2024-09-16T12:20:55.569528+0000","last_became_active":"2024-09-16T12:20:50.381536+0000","last_became_peered":"2024-09-16T12:20:50.381536+0000","last_unstale":"2024-09-16T12:20:55.569528+0000","last_undegraded":"2024-09-16T12:20:55.569528+0000","last_fullsized":"2024-09-16T12:20:55.569528+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:48.519944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:48.519944+0000","last_clean_scrub_stamp":"2024-09-16T12:20:48.519944+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:18.620803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738374,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301660,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070180,"statfs":{"total":96636764160,"available":96327864320,"internally_reserved":0,"allocated":139264,"data_stored":40564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41799999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306404,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065436,"statfs":{"total":96636764160,"available":96323006464,"internally_reserved":0,"allocated":737280,"data_stored":630951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302244,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069596,"statfs":{"total":96636764160,"available":96327266304,"internally_reserved":0,"allocated":737280,"data_stored":630951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T12:20:59.962 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-16T12:20:59.963 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-09-16T12:20:59.963 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-09-16T12:20:59.963 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-09-16T12:21:00.279 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.279 INFO:teuthology.orchestra.run.smithi018.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T12:21:00.292 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-09-16T12:21:00.292 INFO:teuthology.run_tasks:Running task ceph... 2024-09-16T12:21:00.304 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-09-16T12:21:00.304 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-16T12:21:00.305 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907248', 'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'description': 'rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/compare-mirror-images-nbd}', 'email': 'sangadi@redhat.com', 'first_in_suite': False, 'job_id': '7907248', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}], 'name': 'sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}, 'extra_system_packages': ['pv']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, 'owner': 'scheduled_sangadi@teuthology', 'priority': 130, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 9764, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-bharath9-testing-2024-09-13-1444-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'targets': {'smithi018.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi172.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror', 'rbd-nbd'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'extra_system_packages': ['pv']}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.4 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.5 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.6 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/compare_mirror_images.sh']}, 'env': {'RBD_DEVICE_TYPE': 'nbd'}, 'timeout': '3h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-16_10:44:09', 'tube': 'smithi', 'user': 'sangadi', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-16T12:21:00.306 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-16T12:21:00.308 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-16T12:21:00.316 DEBUG:teuthology.orchestra.run.smithi018:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T12:21:00.318 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m0777 -- /var/run/ceph 2024-09-16T12:21:00.376 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:21:00.376 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2024-09-16T12:21:00.381 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-16T12:21:00.381 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2024-09-16T12:21:00.388 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-16T12:21:00.388 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T12:21:00.388 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 349 Links: 1 2024-09-16T12:21:00.389 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T12:21:00.389 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-09-16 12:16:49.336337751 +0000 2024-09-16T12:21:00.389 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-09-16 12:16:49.136342361 +0000 2024-09-16T12:21:00.389 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-09-16 12:16:49.136342361 +0000 2024-09-16T12:21:00.389 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2024-09-16T12:21:00.389 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-16T12:21:00.446 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-09-16T12:21:00.446 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-09-16T12:21:00.446 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000430048 s, 1.2 MB/s 2024-09-16T12:21:00.447 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-16T12:21:00.456 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2024-09-16T12:21:00.462 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-16T12:21:00.462 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T12:21:00.462 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 356 Links: 1 2024-09-16T12:21:00.462 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T12:21:00.462 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-09-16 12:16:49.772327704 +0000 2024-09-16T12:21:00.462 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-09-16 12:16:49.580332129 +0000 2024-09-16T12:21:00.462 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-09-16 12:16:49.580332129 +0000 2024-09-16T12:21:00.463 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2024-09-16T12:21:00.463 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-16T12:21:00.518 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-09-16T12:21:00.518 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-09-16T12:21:00.518 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000400754 s, 1.3 MB/s 2024-09-16T12:21:00.519 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-16T12:21:00.526 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2024-09-16T12:21:00.531 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-16T12:21:00.531 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T12:21:00.531 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 361 Links: 1 2024-09-16T12:21:00.531 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T12:21:00.531 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-09-16 12:16:50.196317933 +0000 2024-09-16T12:21:00.532 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-09-16 12:16:50.000322449 +0000 2024-09-16T12:21:00.532 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-09-16 12:16:50.000322449 +0000 2024-09-16T12:21:00.532 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2024-09-16T12:21:00.532 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-16T12:21:00.547 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-09-16T12:21:00.547 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-09-16T12:21:00.547 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000387109 s, 1.3 MB/s 2024-09-16T12:21:00.548 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-16T12:21:00.600 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2024-09-16T12:21:00.609 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-16T12:21:00.609 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-16T12:21:00.609 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 6h/6d Inode: 369 Links: 1 2024-09-16T12:21:00.609 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-16T12:21:00.609 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-09-16 12:16:50.444312218 +0000 2024-09-16T12:21:00.609 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-09-16 12:16:50.440312309 +0000 2024-09-16T12:21:00.609 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-09-16 12:16:50.440312309 +0000 2024-09-16T12:21:00.609 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: - 2024-09-16T12:21:00.610 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-16T12:21:00.623 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-09-16T12:21:00.623 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-09-16T12:21:00.623 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000377274 s, 1.4 MB/s 2024-09-16T12:21:00.624 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-16T12:21:00.632 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T12:21:00.632 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T12:21:00.632 INFO:tasks.ceph:Generating config... 2024-09-16T12:21:00.634 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-16T12:21:00.634 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-16T12:21:00.634 INFO:tasks.ceph:[client] log to stderr = False 2024-09-16T12:21:00.634 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-16T12:21:00.634 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.634 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.634 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.634 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.634 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.634 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.634 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.634 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.635 INFO:tasks.ceph:[client.mirror.4] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.635 INFO:tasks.ceph:[client.mirror.4] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.635 INFO:tasks.ceph:[client.mirror.5] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.635 INFO:tasks.ceph:[client.mirror.5] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.635 INFO:tasks.ceph:[client.mirror.6] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.635 INFO:tasks.ceph:[client.mirror.6] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.635 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-16T12:21:00.635 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-16T12:21:00.635 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-16T12:21:00.635 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-16T12:21:00.635 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-16T12:21:00.635 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-16T12:21:00.635 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-16T12:21:00.636 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-16T12:21:00.636 INFO:tasks.ceph:[osd] bluefs allocator = hybrid 2024-09-16T12:21:00.636 INFO:tasks.ceph:[osd] bluestore allocator = hybrid 2024-09-16T12:21:00.636 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-16T12:21:00.636 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-16T12:21:00.636 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-16T12:21:00.636 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-16T12:21:00.636 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-16T12:21:00.636 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-16T12:21:00.636 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-16T12:21:00.636 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-16T12:21:00.636 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-16T12:21:00.636 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-16T12:21:00.636 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-16T12:21:00.637 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-16T12:21:00.637 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-09-16T12:21:00.637 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-09-16T12:21:00.706 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/cluster2.keyring 2024-09-16T12:21:00.709 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-09-16T12:21:00.745 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-16T12:21:00.758 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.172')] 2024-09-16T12:21:00.759 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.172', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluefs allocator': 'hybrid', 'bluestore allocator': 'hybrid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.4': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.5': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.6': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-16T12:21:00.760 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:21:00.760 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T12:21:00.806 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.172 --print /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T12:21:00.838 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T12:21:00.838 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: generated fsid ba1e2d9d-316e-497e-977f-9f653ec9cac9 2024-09-16T12:21:00.838 INFO:teuthology.orchestra.run.smithi172.stdout:setting min_mon_release = octopus 2024-09-16T12:21:00.838 INFO:teuthology.orchestra.run.smithi172.stdout:epoch 0 2024-09-16T12:21:00.838 INFO:teuthology.orchestra.run.smithi172.stdout:fsid ba1e2d9d-316e-497e-977f-9f653ec9cac9 2024-09-16T12:21:00.839 INFO:teuthology.orchestra.run.smithi172.stdout:last_changed 2024-09-16T12:21:00.833780+0000 2024-09-16T12:21:00.839 INFO:teuthology.orchestra.run.smithi172.stdout:created 2024-09-16T12:21:00.833780+0000 2024-09-16T12:21:00.839 INFO:teuthology.orchestra.run.smithi172.stdout:min_mon_release 15 (octopus) 2024-09-16T12:21:00.839 INFO:teuthology.orchestra.run.smithi172.stdout:election_strategy: 1 2024-09-16T12:21:00.839 INFO:teuthology.orchestra.run.smithi172.stdout:0: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.a 2024-09-16T12:21:00.839 INFO:teuthology.orchestra.run.smithi172.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-09-16T12:21:00.840 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-16T12:21:00.844 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID ba1e2d9d-316e-497e-977f-9f653ec9cac9... 2024-09-16T12:21:00.845 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-16T12:21:00.847 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-16T12:21:00.879 INFO:teuthology.orchestra.run.smithi018.stdout:[global] 2024-09-16T12:21:00.879 INFO:teuthology.orchestra.run.smithi018.stdout: chdir = "" 2024-09-16T12:21:00.879 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T12:21:00.880 INFO:teuthology.orchestra.run.smithi018.stdout: auth supported = cephx 2024-09-16T12:21:00.880 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.880 INFO:teuthology.orchestra.run.smithi018.stdout: filestore xattr use omap = true 2024-09-16T12:21:00.880 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.880 INFO:teuthology.orchestra.run.smithi018.stdout: mon clock drift allowed = 1.000 2024-09-16T12:21:00.880 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.880 INFO:teuthology.orchestra.run.smithi018.stdout: osd crush chooseleaf type = 0 2024-09-16T12:21:00.880 INFO:teuthology.orchestra.run.smithi018.stdout: auth debug = true 2024-09-16T12:21:00.880 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.880 INFO:teuthology.orchestra.run.smithi018.stdout: ms die on old message = true 2024-09-16T12:21:00.880 INFO:teuthology.orchestra.run.smithi018.stdout: ms die on bug = true 2024-09-16T12:21:00.880 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.880 INFO:teuthology.orchestra.run.smithi018.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T12:21:00.881 INFO:teuthology.orchestra.run.smithi018.stdout: mon pg warn max object skew = 0 2024-09-16T12:21:00.881 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.881 INFO:teuthology.orchestra.run.smithi018.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T12:21:00.881 INFO:teuthology.orchestra.run.smithi018.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T12:21:00.881 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.881 INFO:teuthology.orchestra.run.smithi018.stdout: osd pool default size = 2 2024-09-16T12:21:00.881 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.881 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd allow primary affinity = true 2024-09-16T12:21:00.881 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd allow pg remap = true 2024-09-16T12:21:00.881 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on legacy crush tunables = false 2024-09-16T12:21:00.881 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on crush straw calc version zero = false 2024-09-16T12:21:00.881 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on no sortbitwise = false 2024-09-16T12:21:00.881 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on osd down out interval zero = false 2024-09-16T12:21:00.882 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on too few osds = false 2024-09-16T12:21:00.882 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T12:21:00.882 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T12:21:00.882 INFO:teuthology.orchestra.run.smithi018.stdout: mon_allow_pool_size_one = true 2024-09-16T12:21:00.882 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.882 INFO:teuthology.orchestra.run.smithi018.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T12:21:00.882 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.882 INFO:teuthology.orchestra.run.smithi018.stdout: osd default data pool replay window = 5 2024-09-16T12:21:00.882 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.882 INFO:teuthology.orchestra.run.smithi018.stdout: mon allow pool delete = true 2024-09-16T12:21:00.882 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.882 INFO:teuthology.orchestra.run.smithi018.stdout: mon cluster log file level = debug 2024-09-16T12:21:00.882 INFO:teuthology.orchestra.run.smithi018.stdout: debug asserts on shutdown = true 2024-09-16T12:21:00.883 INFO:teuthology.orchestra.run.smithi018.stdout: mon health detail to clog = false 2024-09-16T12:21:00.883 INFO:teuthology.orchestra.run.smithi018.stdout: mon host = 172.21.15.172 2024-09-16T12:21:00.883 INFO:teuthology.orchestra.run.smithi018.stdout: mon client directed command retry = 5 2024-09-16T12:21:00.883 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on pool no app = False 2024-09-16T12:21:00.883 INFO:teuthology.orchestra.run.smithi018.stdout: ms inject socket failures = 5000 2024-09-16T12:21:00.883 INFO:teuthology.orchestra.run.smithi018.stdout: fsid = ba1e2d9d-316e-497e-977f-9f653ec9cac9 2024-09-16T12:21:00.883 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.883 INFO:teuthology.orchestra.run.smithi018.stdout:[osd] 2024-09-16T12:21:00.883 INFO:teuthology.orchestra.run.smithi018.stdout: osd journal size = 100 2024-09-16T12:21:00.883 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.883 INFO:teuthology.orchestra.run.smithi018.stdout: osd scrub load threshold = 5.0 2024-09-16T12:21:00.883 INFO:teuthology.orchestra.run.smithi018.stdout: osd scrub max interval = 600 2024-09-16T12:21:00.883 INFO:teuthology.orchestra.run.smithi018.stdout: osd mclock profile = high_recovery_ops 2024-09-16T12:21:00.884 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.884 INFO:teuthology.orchestra.run.smithi018.stdout: osd recover clone overlap = true 2024-09-16T12:21:00.884 INFO:teuthology.orchestra.run.smithi018.stdout: osd recovery max chunk = 1048576 2024-09-16T12:21:00.884 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.884 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug shutdown = true 2024-09-16T12:21:00.884 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug op order = true 2024-09-16T12:21:00.884 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug verify stray on activate = true 2024-09-16T12:21:00.884 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.884 INFO:teuthology.orchestra.run.smithi018.stdout: osd open classes on start = true 2024-09-16T12:21:00.884 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug pg log writeout = true 2024-09-16T12:21:00.884 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.884 INFO:teuthology.orchestra.run.smithi018.stdout: osd deep scrub update digest min age = 30 2024-09-16T12:21:00.884 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.885 INFO:teuthology.orchestra.run.smithi018.stdout: osd map max advance = 10 2024-09-16T12:21:00.885 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.885 INFO:teuthology.orchestra.run.smithi018.stdout: journal zero on create = true 2024-09-16T12:21:00.885 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.885 INFO:teuthology.orchestra.run.smithi018.stdout: filestore ondisk finisher threads = 3 2024-09-16T12:21:00.885 INFO:teuthology.orchestra.run.smithi018.stdout: filestore apply finisher threads = 3 2024-09-16T12:21:00.885 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.885 INFO:teuthology.orchestra.run.smithi018.stdout: bdev debug aio = true 2024-09-16T12:21:00.885 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug misdirected ops = true 2024-09-16T12:21:00.885 INFO:teuthology.orchestra.run.smithi018.stdout: bluefs allocator = hybrid 2024-09-16T12:21:00.885 INFO:teuthology.orchestra.run.smithi018.stdout: bluestore allocator = hybrid 2024-09-16T12:21:00.885 INFO:teuthology.orchestra.run.smithi018.stdout: bluestore block size = 96636764160 2024-09-16T12:21:00.885 INFO:teuthology.orchestra.run.smithi018.stdout: bluestore fsck on mount = True 2024-09-16T12:21:00.886 INFO:teuthology.orchestra.run.smithi018.stdout: debug bluefs = 1/20 2024-09-16T12:21:00.886 INFO:teuthology.orchestra.run.smithi018.stdout: debug bluestore = 1/20 2024-09-16T12:21:00.886 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2024-09-16T12:21:00.886 INFO:teuthology.orchestra.run.smithi018.stdout: debug osd = 20 2024-09-16T12:21:00.886 INFO:teuthology.orchestra.run.smithi018.stdout: debug rocksdb = 4/10 2024-09-16T12:21:00.886 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T12:21:00.886 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd full ratio = 0.9 2024-09-16T12:21:00.886 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd nearfull ratio = 0.8 2024-09-16T12:21:00.886 INFO:teuthology.orchestra.run.smithi018.stdout: osd failsafe full ratio = 0.95 2024-09-16T12:21:00.886 INFO:teuthology.orchestra.run.smithi018.stdout: osd objectstore = bluestore 2024-09-16T12:21:00.886 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.886 INFO:teuthology.orchestra.run.smithi018.stdout:[mgr] 2024-09-16T12:21:00.886 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2024-09-16T12:21:00.887 INFO:teuthology.orchestra.run.smithi018.stdout: debug mgr = 20 2024-09-16T12:21:00.887 INFO:teuthology.orchestra.run.smithi018.stdout: debug mon = 20 2024-09-16T12:21:00.887 INFO:teuthology.orchestra.run.smithi018.stdout: debug auth = 20 2024-09-16T12:21:00.887 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min pgs per osd = 4 2024-09-16T12:21:00.887 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min bytes per osd = 10 2024-09-16T12:21:00.887 INFO:teuthology.orchestra.run.smithi018.stdout: mgr/telemetry/nag = false 2024-09-16T12:21:00.887 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.887 INFO:teuthology.orchestra.run.smithi018.stdout:[mon] 2024-09-16T12:21:00.887 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2024-09-16T12:21:00.887 INFO:teuthology.orchestra.run.smithi018.stdout: debug mon = 20 2024-09-16T12:21:00.887 INFO:teuthology.orchestra.run.smithi018.stdout: debug paxos = 20 2024-09-16T12:21:00.887 INFO:teuthology.orchestra.run.smithi018.stdout: debug auth = 20 2024-09-16T12:21:00.887 INFO:teuthology.orchestra.run.smithi018.stdout: mon data avail warn = 5 2024-09-16T12:21:00.888 INFO:teuthology.orchestra.run.smithi018.stdout: mon mgr mkfs grace = 240 2024-09-16T12:21:00.888 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min pgs per osd = 4 2024-09-16T12:21:00.888 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd reporter subtree level = osd 2024-09-16T12:21:00.888 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd prime pg temp = true 2024-09-16T12:21:00.888 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min bytes per osd = 10 2024-09-16T12:21:00.888 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.888 INFO:teuthology.orchestra.run.smithi018.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T12:21:00.888 INFO:teuthology.orchestra.run.smithi018.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T12:21:00.888 INFO:teuthology.orchestra.run.smithi018.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T12:21:00.888 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.888 INFO:teuthology.orchestra.run.smithi018.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T12:21:00.888 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T12:21:00.888 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T12:21:00.889 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.889 INFO:teuthology.orchestra.run.smithi018.stdout: # 1m isn't quite enough 2024-09-16T12:21:00.889 INFO:teuthology.orchestra.run.smithi018.stdout: mon_down_mkfs_grace = 2m 2024-09-16T12:21:00.889 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.889 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_filestore_osds = false 2024-09-16T12:21:00.889 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T12:21:00.889 INFO:teuthology.orchestra.run.smithi018.stdout:[client] 2024-09-16T12:21:00.889 INFO:teuthology.orchestra.run.smithi018.stdout: rgw cache enabled = true 2024-09-16T12:21:00.889 INFO:teuthology.orchestra.run.smithi018.stdout: rgw enable ops log = true 2024-09-16T12:21:00.889 INFO:teuthology.orchestra.run.smithi018.stdout: rgw enable usage log = true 2024-09-16T12:21:00.889 INFO:teuthology.orchestra.run.smithi018.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T12:21:00.889 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T12:21:00.889 INFO:teuthology.orchestra.run.smithi018.stdout: debug rbd = 20 2024-09-16T12:21:00.889 INFO:teuthology.orchestra.run.smithi018.stdout: debug rbd_mirror = 15 2024-09-16T12:21:00.890 INFO:teuthology.orchestra.run.smithi018.stdout: log to stderr = False 2024-09-16T12:21:00.890 INFO:teuthology.orchestra.run.smithi018.stdout: rbd default features = 125 2024-09-16T12:21:00.890 INFO:teuthology.orchestra.run.smithi018.stdout:[mon.a] 2024-09-16T12:21:00.890 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.0] 2024-09-16T12:21:00.890 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.890 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.890 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.1] 2024-09-16T12:21:00.890 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.890 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.890 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.2] 2024-09-16T12:21:00.890 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.890 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.890 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.3] 2024-09-16T12:21:00.890 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.891 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.891 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.4] 2024-09-16T12:21:00.891 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.891 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.891 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.5] 2024-09-16T12:21:00.891 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.891 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.891 INFO:teuthology.orchestra.run.smithi018.stdout:[client.mirror.6] 2024-09-16T12:21:00.891 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.891 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.893 INFO:teuthology.orchestra.run.smithi172.stdout:[global] 2024-09-16T12:21:00.893 INFO:teuthology.orchestra.run.smithi172.stdout: chdir = "" 2024-09-16T12:21:00.893 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-16T12:21:00.893 INFO:teuthology.orchestra.run.smithi172.stdout: auth supported = cephx 2024-09-16T12:21:00.893 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.893 INFO:teuthology.orchestra.run.smithi172.stdout: filestore xattr use omap = true 2024-09-16T12:21:00.893 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.893 INFO:teuthology.orchestra.run.smithi172.stdout: mon clock drift allowed = 1.000 2024-09-16T12:21:00.893 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.893 INFO:teuthology.orchestra.run.smithi172.stdout: osd crush chooseleaf type = 0 2024-09-16T12:21:00.893 INFO:teuthology.orchestra.run.smithi172.stdout: auth debug = true 2024-09-16T12:21:00.893 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.893 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on old message = true 2024-09-16T12:21:00.894 INFO:teuthology.orchestra.run.smithi172.stdout: ms die on bug = true 2024-09-16T12:21:00.894 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.894 INFO:teuthology.orchestra.run.smithi172.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-16T12:21:00.894 INFO:teuthology.orchestra.run.smithi172.stdout: mon pg warn max object skew = 0 2024-09-16T12:21:00.894 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.894 INFO:teuthology.orchestra.run.smithi172.stdout: # disable pg_autoscaler by default for new pools 2024-09-16T12:21:00.894 INFO:teuthology.orchestra.run.smithi172.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-16T12:21:00.894 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.894 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default size = 2 2024-09-16T12:21:00.894 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.894 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow primary affinity = true 2024-09-16T12:21:00.894 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd allow pg remap = true 2024-09-16T12:21:00.894 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on legacy crush tunables = false 2024-09-16T12:21:00.895 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on crush straw calc version zero = false 2024-09-16T12:21:00.895 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on no sortbitwise = false 2024-09-16T12:21:00.895 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on osd down out interval zero = false 2024-09-16T12:21:00.895 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on too few osds = false 2024-09-16T12:21:00.895 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-16T12:21:00.895 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-16T12:21:00.895 INFO:teuthology.orchestra.run.smithi172.stdout: mon_allow_pool_size_one = true 2024-09-16T12:21:00.895 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.895 INFO:teuthology.orchestra.run.smithi172.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-16T12:21:00.895 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.895 INFO:teuthology.orchestra.run.smithi172.stdout: osd default data pool replay window = 5 2024-09-16T12:21:00.895 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.895 INFO:teuthology.orchestra.run.smithi172.stdout: mon allow pool delete = true 2024-09-16T12:21:00.895 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.896 INFO:teuthology.orchestra.run.smithi172.stdout: mon cluster log file level = debug 2024-09-16T12:21:00.896 INFO:teuthology.orchestra.run.smithi172.stdout: debug asserts on shutdown = true 2024-09-16T12:21:00.896 INFO:teuthology.orchestra.run.smithi172.stdout: mon health detail to clog = false 2024-09-16T12:21:00.896 INFO:teuthology.orchestra.run.smithi172.stdout: mon host = 172.21.15.172 2024-09-16T12:21:00.896 INFO:teuthology.orchestra.run.smithi172.stdout: mon client directed command retry = 5 2024-09-16T12:21:00.896 INFO:teuthology.orchestra.run.smithi172.stdout: mon warn on pool no app = False 2024-09-16T12:21:00.896 INFO:teuthology.orchestra.run.smithi172.stdout: ms inject socket failures = 5000 2024-09-16T12:21:00.896 INFO:teuthology.orchestra.run.smithi172.stdout: fsid = ba1e2d9d-316e-497e-977f-9f653ec9cac9 2024-09-16T12:21:00.896 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.896 INFO:teuthology.orchestra.run.smithi172.stdout:[osd] 2024-09-16T12:21:00.896 INFO:teuthology.orchestra.run.smithi172.stdout: osd journal size = 100 2024-09-16T12:21:00.896 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.896 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub load threshold = 5.0 2024-09-16T12:21:00.896 INFO:teuthology.orchestra.run.smithi172.stdout: osd scrub max interval = 600 2024-09-16T12:21:00.897 INFO:teuthology.orchestra.run.smithi172.stdout: osd mclock profile = high_recovery_ops 2024-09-16T12:21:00.897 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.897 INFO:teuthology.orchestra.run.smithi172.stdout: osd recover clone overlap = true 2024-09-16T12:21:00.897 INFO:teuthology.orchestra.run.smithi172.stdout: osd recovery max chunk = 1048576 2024-09-16T12:21:00.897 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.897 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug shutdown = true 2024-09-16T12:21:00.897 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug op order = true 2024-09-16T12:21:00.897 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug verify stray on activate = true 2024-09-16T12:21:00.897 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.897 INFO:teuthology.orchestra.run.smithi172.stdout: osd open classes on start = true 2024-09-16T12:21:00.897 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug pg log writeout = true 2024-09-16T12:21:00.898 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.898 INFO:teuthology.orchestra.run.smithi172.stdout: osd deep scrub update digest min age = 30 2024-09-16T12:21:00.898 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.898 INFO:teuthology.orchestra.run.smithi172.stdout: osd map max advance = 10 2024-09-16T12:21:00.898 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.898 INFO:teuthology.orchestra.run.smithi172.stdout: journal zero on create = true 2024-09-16T12:21:00.898 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.898 INFO:teuthology.orchestra.run.smithi172.stdout: filestore ondisk finisher threads = 3 2024-09-16T12:21:00.898 INFO:teuthology.orchestra.run.smithi172.stdout: filestore apply finisher threads = 3 2024-09-16T12:21:00.898 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.898 INFO:teuthology.orchestra.run.smithi172.stdout: bdev debug aio = true 2024-09-16T12:21:00.898 INFO:teuthology.orchestra.run.smithi172.stdout: osd debug misdirected ops = true 2024-09-16T12:21:00.898 INFO:teuthology.orchestra.run.smithi172.stdout: bluefs allocator = hybrid 2024-09-16T12:21:00.899 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore allocator = hybrid 2024-09-16T12:21:00.899 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore block size = 96636764160 2024-09-16T12:21:00.899 INFO:teuthology.orchestra.run.smithi172.stdout: bluestore fsck on mount = True 2024-09-16T12:21:00.899 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluefs = 1/20 2024-09-16T12:21:00.899 INFO:teuthology.orchestra.run.smithi172.stdout: debug bluestore = 1/20 2024-09-16T12:21:00.899 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-09-16T12:21:00.899 INFO:teuthology.orchestra.run.smithi172.stdout: debug osd = 20 2024-09-16T12:21:00.899 INFO:teuthology.orchestra.run.smithi172.stdout: debug rocksdb = 4/10 2024-09-16T12:21:00.899 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd backfillfull_ratio = 0.85 2024-09-16T12:21:00.899 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd full ratio = 0.9 2024-09-16T12:21:00.899 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd nearfull ratio = 0.8 2024-09-16T12:21:00.899 INFO:teuthology.orchestra.run.smithi172.stdout: osd failsafe full ratio = 0.95 2024-09-16T12:21:00.899 INFO:teuthology.orchestra.run.smithi172.stdout: osd objectstore = bluestore 2024-09-16T12:21:00.899 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.900 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr] 2024-09-16T12:21:00.900 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-09-16T12:21:00.900 INFO:teuthology.orchestra.run.smithi172.stdout: debug mgr = 20 2024-09-16T12:21:00.900 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2024-09-16T12:21:00.900 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2024-09-16T12:21:00.900 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2024-09-16T12:21:00.900 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2024-09-16T12:21:00.900 INFO:teuthology.orchestra.run.smithi172.stdout: mgr/telemetry/nag = false 2024-09-16T12:21:00.900 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.900 INFO:teuthology.orchestra.run.smithi172.stdout:[mon] 2024-09-16T12:21:00.900 INFO:teuthology.orchestra.run.smithi172.stdout: debug ms = 1 2024-09-16T12:21:00.900 INFO:teuthology.orchestra.run.smithi172.stdout: debug mon = 20 2024-09-16T12:21:00.900 INFO:teuthology.orchestra.run.smithi172.stdout: debug paxos = 20 2024-09-16T12:21:00.900 INFO:teuthology.orchestra.run.smithi172.stdout: debug auth = 20 2024-09-16T12:21:00.901 INFO:teuthology.orchestra.run.smithi172.stdout: mon data avail warn = 5 2024-09-16T12:21:00.901 INFO:teuthology.orchestra.run.smithi172.stdout: mon mgr mkfs grace = 240 2024-09-16T12:21:00.901 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min pgs per osd = 4 2024-09-16T12:21:00.901 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd reporter subtree level = osd 2024-09-16T12:21:00.901 INFO:teuthology.orchestra.run.smithi172.stdout: mon osd prime pg temp = true 2024-09-16T12:21:00.901 INFO:teuthology.orchestra.run.smithi172.stdout: mon reweight min bytes per osd = 10 2024-09-16T12:21:00.901 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.901 INFO:teuthology.orchestra.run.smithi172.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-16T12:21:00.901 INFO:teuthology.orchestra.run.smithi172.stdout: auth mon ticket ttl = 660 # 11m 2024-09-16T12:21:00.901 INFO:teuthology.orchestra.run.smithi172.stdout: auth service ticket ttl = 240 # 4m 2024-09-16T12:21:00.901 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.901 INFO:teuthology.orchestra.run.smithi172.stdout: # don't complain about insecure global_id in the test suite 2024-09-16T12:21:00.901 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-16T12:21:00.901 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-16T12:21:00.902 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.902 INFO:teuthology.orchestra.run.smithi172.stdout: # 1m isn't quite enough 2024-09-16T12:21:00.902 INFO:teuthology.orchestra.run.smithi172.stdout: mon_down_mkfs_grace = 2m 2024-09-16T12:21:00.902 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.902 INFO:teuthology.orchestra.run.smithi172.stdout: mon_warn_on_filestore_osds = false 2024-09-16T12:21:00.902 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:00.902 INFO:teuthology.orchestra.run.smithi172.stdout:[client] 2024-09-16T12:21:00.902 INFO:teuthology.orchestra.run.smithi172.stdout: rgw cache enabled = true 2024-09-16T12:21:00.902 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable ops log = true 2024-09-16T12:21:00.902 INFO:teuthology.orchestra.run.smithi172.stdout: rgw enable usage log = true 2024-09-16T12:21:00.902 INFO:teuthology.orchestra.run.smithi172.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-16T12:21:00.902 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-16T12:21:00.902 INFO:teuthology.orchestra.run.smithi172.stdout: debug rbd = 20 2024-09-16T12:21:00.902 INFO:teuthology.orchestra.run.smithi172.stdout: debug rbd_mirror = 15 2024-09-16T12:21:00.903 INFO:teuthology.orchestra.run.smithi172.stdout: log to stderr = False 2024-09-16T12:21:00.903 INFO:teuthology.orchestra.run.smithi172.stdout: rbd default features = 125 2024-09-16T12:21:00.903 INFO:teuthology.orchestra.run.smithi172.stdout:[mon.a] 2024-09-16T12:21:00.903 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.0] 2024-09-16T12:21:00.903 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.903 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.903 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.1] 2024-09-16T12:21:00.903 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.903 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.903 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.2] 2024-09-16T12:21:00.903 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.903 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.903 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.3] 2024-09-16T12:21:00.904 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.904 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.904 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.4] 2024-09-16T12:21:00.904 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.904 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.904 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.5] 2024-09-16T12:21:00.904 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.904 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.904 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.6] 2024-09-16T12:21:00.904 INFO:teuthology.orchestra.run.smithi172.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-16T12:21:00.904 INFO:teuthology.orchestra.run.smithi172.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-16T12:21:00.904 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-09-16T12:21:00.905 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-09-16T12:21:00.983 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-16T12:21:00.983 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:21:00.983 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-09-16T12:21:00.990 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:21:00.990 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-09-16T12:21:00.996 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi018.front.sepia.ceph.com 2024-09-16T12:21:00.996 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:21:00.996 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-16T12:21:00.996 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-16T12:21:01.023 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:21:01.023 DEBUG:teuthology.orchestra.run.smithi018:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-16T12:21:01.030 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi172.front.sepia.ceph.com 2024-09-16T12:21:01.030 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:21:01.030 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-16T12:21:01.030 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-16T12:21:01.066 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:21:01.066 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-16T12:21:01.072 INFO:tasks.ceph:Setting up mon nodes... 2024-09-16T12:21:01.073 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-16T12:21:01.073 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-16T12:21:01.119 INFO:teuthology.orchestra.run.smithi018.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-16T12:21:01.122 INFO:tasks.ceph:Setting up mds nodes... 2024-09-16T12:21:01.122 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-16T12:21:01.123 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-09-16T12:21:01.156 INFO:teuthology.orchestra.run.smithi018.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-09-16T12:21:01.168 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-09-16T12:21:01.205 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-09-16T12:21:01.218 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-16T12:21:01.247 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-16T12:21:01.260 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-16T12:21:01.289 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-16T12:21:01.300 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-16T12:21:01.381 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-16T12:21:01.392 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-16T12:21:01.422 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-16T12:21:01.432 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-16T12:21:01.465 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-16T12:21:01.477 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-16T12:21:01.507 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-16T12:21:01.520 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-16T12:21:01.599 INFO:teuthology.orchestra.run.smithi172.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-16T12:21:01.611 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-16T12:21:01.611 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi018.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi172.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-16T12:21:01.611 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-09-16T12:21:01.622 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T12:21:01.622 INFO:tasks.ceph:role: cluster2.osd.0 2024-09-16T12:21:01.622 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com 2024-09-16T12:21:01.622 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-16T12:21:02.145 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T12:21:02.145 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T12:21:02.145 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T12:21:02.145 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 2024-09-16T12:21:02.145 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T12:21:02.145 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-09-16T12:21:02.145 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T12:21:02.145 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T12:21:02.146 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T12:21:02.146 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T12:21:02.147 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-09-16T12:21:02.147 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-09-16T12:21:02.240 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-09-16T12:21:02.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:21:02.252 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T12:21:02.253 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-09-16T12:21:02.311 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T12:21:02.311 INFO:tasks.ceph:role: cluster2.osd.1 2024-09-16T12:21:02.311 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com 2024-09-16T12:21:02.311 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-16T12:21:02.829 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T12:21:02.830 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T12:21:02.830 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T12:21:02.830 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 2024-09-16T12:21:02.830 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T12:21:02.830 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-09-16T12:21:02.830 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T12:21:02.830 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T12:21:02.830 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T12:21:02.830 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T12:21:02.831 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-09-16T12:21:02.832 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-09-16T12:21:02.871 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-09-16T12:21:02.923 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T12:21:02.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:21:02.925 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-09-16T12:21:02.934 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-16T12:21:02.935 INFO:tasks.ceph:role: cluster2.osd.2 2024-09-16T12:21:02.935 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com 2024-09-16T12:21:02.935 DEBUG:teuthology.orchestra.run.smithi172:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-16T12:21:03.409 INFO:teuthology.orchestra.run.smithi172.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-16T12:21:03.409 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-16T12:21:03.409 INFO:teuthology.orchestra.run.smithi172.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-16T12:21:03.409 INFO:teuthology.orchestra.run.smithi172.stdout: = reflink=1 2024-09-16T12:21:03.409 INFO:teuthology.orchestra.run.smithi172.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-16T12:21:03.410 INFO:teuthology.orchestra.run.smithi172.stdout: = sunit=0 swidth=0 blks 2024-09-16T12:21:03.410 INFO:teuthology.orchestra.run.smithi172.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-16T12:21:03.410 INFO:teuthology.orchestra.run.smithi172.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-16T12:21:03.410 INFO:teuthology.orchestra.run.smithi172.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-16T12:21:03.410 INFO:teuthology.orchestra.run.smithi172.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-16T12:21:03.411 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi172.front.sepia.ceph.com -o noatime 2024-09-16T12:21:03.411 DEBUG:teuthology.orchestra.run.smithi172:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-09-16T12:21:03.469 DEBUG:teuthology.orchestra.run.smithi172:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-09-16T12:21:03.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:21:03.481 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: /sbin/restorecon: command not found 2024-09-16T12:21:03.481 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T12:21:03.543 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:03.533+0000 7fe1d6fcd540 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-09-16T12:21:03.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:03.537+0000 7fe1d6fcd540 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-09-16T12:21:03.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:03.537+0000 7fe1d6fcd540 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-09-16T12:21:03.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:03.537+0000 7fe1d6fcd540 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-09-16T12:21:03.546 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:03.537+0000 7fe1d6fcd540 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2024-09-16T12:21:03.550 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:03.541+0000 7fe1d6fcd540 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-09-16T12:21:04.113 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:04.105+0000 7fe1d6fcd540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:21:04.906 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:04.897+0000 7fe1d6fcd540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:21:05.687 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:05.677+0000 7fe1d6fcd540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:21:06.540 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-09-16T12:21:06.557 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T12:21:06.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:06.629+0000 7f48f7596540 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-09-16T12:21:06.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:06.629+0000 7f48f7596540 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-09-16T12:21:06.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:06.629+0000 7f48f7596540 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-09-16T12:21:06.640 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:06.629+0000 7f48f7596540 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-09-16T12:21:06.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:06.629+0000 7f48f7596540 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2024-09-16T12:21:06.641 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:06.633+0000 7f48f7596540 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-09-16T12:21:07.196 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:07.185+0000 7f48f7596540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:21:07.986 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:07.977+0000 7f48f7596540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:21:08.769 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:08.761+0000 7f48f7596540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:21:09.612 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-09-16T12:21:09.628 DEBUG:teuthology.orchestra.run.smithi172:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T12:21:09.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:09.705+0000 7f129bb3c540 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-09-16T12:21:09.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:09.705+0000 7f129bb3c540 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-09-16T12:21:09.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:09.705+0000 7f129bb3c540 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-09-16T12:21:09.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:09.705+0000 7f129bb3c540 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-09-16T12:21:09.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:09.705+0000 7f129bb3c540 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2024-09-16T12:21:09.713 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:09.705+0000 7f129bb3c540 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-09-16T12:21:10.265 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:10.257+0000 7f129bb3c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:21:11.050 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:11.041+0000 7f129bb3c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:21:11.838 INFO:teuthology.orchestra.run.smithi172.stderr:2024-09-16T12:21:11.829+0000 7f129bb3c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-16T12:21:12.688 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-09-16T12:21:12.704 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-16T12:21:12.704 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:21:12.704 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-09-16T12:21:12.721 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:21:12.721 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-09-16T12:21:12.737 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:21:12.737 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-09-16T12:21:12.794 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:21:12.795 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-09-16T12:21:12.804 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-09-16T12:21:12.804 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-09-16T12:21:12.811 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:21:12.811 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-09-16T12:21:12.818 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:21:12.818 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-09-16T12:21:12.866 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:21:12.866 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-09-16T12:21:12.873 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:21:12.873 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-09-16T12:21:12.880 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:21:12.880 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-09-16T12:21:12.886 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:21:12.886 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-09-16T12:21:12.892 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:21:12.892 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-09-16T12:21:12.898 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:21:12.898 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-09-16T12:21:12.946 INFO:tasks.ceph:Adding keys to all mons... 2024-09-16T12:21:12.946 DEBUG:teuthology.orchestra.run.smithi172:> sudo tee -a /etc/ceph/cluster2.keyring 2024-09-16T12:21:12.961 INFO:teuthology.orchestra.run.smithi172.stdout:[mgr.x] 2024-09-16T12:21:12.961 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCtIuhm9NvtBhAAApdctyIVx646vdoso3bFeA== 2024-09-16T12:21:12.962 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.0] 2024-09-16T12:21:12.962 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCvIuhmXSQdIBAA1a3RlO+7Jd/Gawi/VxJC2w== 2024-09-16T12:21:12.962 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.1] 2024-09-16T12:21:12.962 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCyIuhmN2PZJRAAbL2qQPZIZQW0ypxfveTBUQ== 2024-09-16T12:21:12.962 INFO:teuthology.orchestra.run.smithi172.stdout:[osd.2] 2024-09-16T12:21:12.962 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQC1IuhmJScuKhAA1MVCj5rt34kTnJIDEvaLLw== 2024-09-16T12:21:12.962 INFO:teuthology.orchestra.run.smithi172.stdout:[client.0] 2024-09-16T12:21:12.962 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCtIuhm+pIVCRAAL3cLjmq+vZTEHCQpTDNj5w== 2024-09-16T12:21:12.962 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror] 2024-09-16T12:21:12.962 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCtIuhmEZDyCxAApibiMkWqCxLTk2YlsT7tEQ== 2024-09-16T12:21:12.962 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.0] 2024-09-16T12:21:12.962 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCtIuhmVxV8DhAAP70kD9pWTyMXZA++Zg9IcA== 2024-09-16T12:21:12.962 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.1] 2024-09-16T12:21:12.963 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCtIuhm+nD8EBAAcDDb9tYqxipjqDMKfdtXpw== 2024-09-16T12:21:12.963 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.2] 2024-09-16T12:21:12.963 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCtIuhmabNbFhAAwDWtfdiAOXzc8AZZX4Wsyw== 2024-09-16T12:21:12.963 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.3] 2024-09-16T12:21:12.963 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCtIuhmNCDqGBAAxVLeen090e32AQlzkBU7bg== 2024-09-16T12:21:12.963 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.4] 2024-09-16T12:21:12.963 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCtIuhmEIh/GxAA1g2BxntBLT/nPyLi3zLqYA== 2024-09-16T12:21:12.963 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.5] 2024-09-16T12:21:12.963 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCtIuhmvCL9HRAA7gPiRi8G2WKgHCiUL8rtGg== 2024-09-16T12:21:12.963 INFO:teuthology.orchestra.run.smithi172.stdout:[client.mirror.6] 2024-09-16T12:21:12.963 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQCtIuhm4jluIxAAJw82vzu90JPDhU6nIG/0nQ== 2024-09-16T12:21:12.964 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-16T12:21:13.004 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T12:21:13.094 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T12:21:13.137 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-16T12:21:13.177 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:21:13.267 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:21:13.311 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:21:13.354 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:21:13.444 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:21:13.491 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:21:13.542 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:21:13.634 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:21:13.677 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-16T12:21:13.721 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-16T12:21:13.722 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-09-16T12:21:13.735 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-09-16T12:21:13.780 DEBUG:teuthology.orchestra.run.smithi172:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-09-16T12:21:13.841 DEBUG:teuthology.orchestra.run.smithi172:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-09-16T12:21:13.847 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-09-16T12:21:13.847 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-09-16T12:21:13.848 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-09-16T12:21:13.850 INFO:tasks.ceph.cluster2.mon.a:Started 2024-09-16T12:21:13.850 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-09-16T12:21:13.850 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-09-16T12:21:13.850 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-09-16T12:21:13.853 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-09-16T12:21:13.853 INFO:tasks.ceph:Setting crush tunables to default 2024-09-16T12:21:13.854 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster cluster2 osd crush tunables default 2024-09-16T12:21:14.093 INFO:teuthology.orchestra.run.smithi172.stderr:adjusted tunables profile to default 2024-09-16T12:21:14.104 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-09-16T12:21:14.105 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:21:14.105 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-09-16T12:21:14.115 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:21:14.115 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-09-16T12:21:14.125 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:21:14.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-09-16T12:21:14.135 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster cluster2 osd new 2dfcac63-8190-45d1-983a-dcb5bf33d348 0 2024-09-16T12:21:14.227 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:14.222+0000 7ff7b5c74000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-16T12:21:14.329 INFO:teuthology.orchestra.run.smithi172.stdout:0 2024-09-16T12:21:14.342 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster cluster2 osd new 7e338fac-a366-462f-90e5-f16f57b3706f 1 2024-09-16T12:21:14.543 INFO:teuthology.orchestra.run.smithi172.stdout:1 2024-09-16T12:21:14.556 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster cluster2 osd new 5987d4f0-55de-4238-adc4-ca6c9b283f18 2 2024-09-16T12:21:14.662 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:14.654+0000 7ff7b5c74000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-16T12:21:14.713 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:14.706+0000 7ff7b5c74000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-16T12:21:14.756 INFO:teuthology.orchestra.run.smithi172.stdout:2 2024-09-16T12:21:14.769 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-09-16T12:21:14.769 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-09-16T12:21:14.772 INFO:tasks.ceph.cluster2.osd.0:Started 2024-09-16T12:21:14.772 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-09-16T12:21:14.772 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-09-16T12:21:14.774 INFO:tasks.ceph.cluster2.osd.1:Started 2024-09-16T12:21:14.774 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-09-16T12:21:14.774 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-09-16T12:21:14.776 INFO:tasks.ceph.cluster2.osd.2:Started 2024-09-16T12:21:14.776 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T12:21:14.822 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:14.814+0000 7ff7b5c74000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-16T12:21:15.025 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:15.018+0000 7ff7b5c74000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-16T12:21:15.075 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:15.070+0000 7ff7b5c74000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-16T12:21:15.146 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:21:15.137+0000 7f74ef7e2540 -1 Falling back to public interface 2024-09-16T12:21:15.167 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:15.168 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":5,"fsid":"ba1e2d9d-316e-497e-977f-9f653ec9cac9","created":"2024-09-16T12:21:13.976545+0000","modified":"2024-09-16T12:21:14.747781+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T12:21:14.747781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"2dfcac63-8190-45d1-983a-dcb5bf33d348","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7e338fac-a366-462f-90e5-f16f57b3706f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5987d4f0-55de-4238-adc4-ca6c9b283f18","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:21:15.181 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-09-16T12:21:15.181 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-16T12:21:15.230 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:15.222+0000 7ff7b5c74000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-16T12:21:15.288 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:15.282+0000 7ff7b5c74000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-16T12:21:15.348 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:15.342+0000 7ff7b5c74000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-16T12:21:15.370 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T12:21:15.361+0000 7fcf06866540 -1 Falling back to public interface 2024-09-16T12:21:15.374 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:21:15.365+0000 7f3170864540 -1 Falling back to public interface 2024-09-16T12:21:15.484 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-16T12:21:15.653 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:15.646+0000 7ff7b5c74000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-16T12:21:15.673 INFO:teuthology.misc.health.smithi172.stdout: 2024-09-16T12:21:15.673 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":5,"fsid":"ba1e2d9d-316e-497e-977f-9f653ec9cac9","created":"2024-09-16T12:21:13.976545+0000","modified":"2024-09-16T12:21:14.747781+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T12:21:14.747781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"2dfcac63-8190-45d1-983a-dcb5bf33d348","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7e338fac-a366-462f-90e5-f16f57b3706f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5987d4f0-55de-4238-adc4-ca6c9b283f18","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:21:15.684 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T12:21:15.766 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:15.758+0000 7ff7b5c74000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-16T12:21:16.057 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:16.050+0000 7ff7b5c74000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-16T12:21:16.224 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:16.218+0000 7ff7b5c74000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-16T12:21:16.290 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:16.282+0000 7ff7b5c74000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-16T12:21:16.344 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:16.338+0000 7ff7b5c74000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-16T12:21:16.401 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:16.394+0000 7ff7b5c74000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-16T12:21:16.576 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:16.570+0000 7ff7b5c74000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-16T12:21:16.631 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:16.626+0000 7ff7b5c74000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-16T12:21:16.694 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:16.686+0000 7ff7b5c74000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-16T12:21:16.849 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:16.842+0000 7ff7b5c74000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-16T12:21:16.913 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:16.906+0000 7ff7b5c74000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-16T12:21:17.524 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:17.518+0000 7ff7b5c74000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-16T12:21:18.558 INFO:tasks.ceph.cluster2.mgr.x.smithi018.stderr:2024-09-16T12:21:18.550+0000 7ff79ef28700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-16T12:21:18.867 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T12:21:18.856+0000 7fcf06866540 -1 osd.1 0 log_to_monitors true 2024-09-16T12:21:18.938 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:21:18.928+0000 7f74ef7e2540 -1 osd.0 0 log_to_monitors true 2024-09-16T12:21:19.086 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:21:19.076+0000 7f3170864540 -1 osd.2 0 log_to_monitors true 2024-09-16T12:21:21.987 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-16T12:21:22.370 INFO:teuthology.misc.health.smithi172.stdout: 2024-09-16T12:21:22.370 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":7,"fsid":"ba1e2d9d-316e-497e-977f-9f653ec9cac9","created":"2024-09-16T12:21:13.976545+0000","modified":"2024-09-16T12:21:20.607793+0000","last_up_change":"0.000000","last_in_change":"2024-09-16T12:21:14.747781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"2dfcac63-8190-45d1-983a-dcb5bf33d348","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7e338fac-a366-462f-90e5-f16f57b3706f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5987d4f0-55de-4238-adc4-ca6c9b283f18","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:21:22.386 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-16T12:21:22.651 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T12:21:22.640+0000 7fcf02006700 -1 osd.1 0 waiting for initial osdmap 2024-09-16T12:21:22.657 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T12:21:22.648+0000 7fcefada9700 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T12:21:22.675 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:21:22.664+0000 7f316c004700 -1 osd.2 0 waiting for initial osdmap 2024-09-16T12:21:22.682 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:21:22.672+0000 7f3164da7700 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T12:21:22.694 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:21:22.684+0000 7f74eaf82700 -1 osd.0 0 waiting for initial osdmap 2024-09-16T12:21:22.701 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:21:22.692+0000 7f74e3d25700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-16T12:21:28.690 DEBUG:teuthology.orchestra.run.smithi172:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-16T12:21:28.978 INFO:teuthology.misc.health.smithi172.stdout: 2024-09-16T12:21:28.979 INFO:teuthology.misc.health.smithi172.stdout:{"epoch":12,"fsid":"ba1e2d9d-316e-497e-977f-9f653ec9cac9","created":"2024-09-16T12:21:13.976545+0000","modified":"2024-09-16T12:21:27.685341+0000","last_up_change":"2024-09-16T12:21:23.671417+0000","last_in_change":"2024-09-16T12:21:14.747781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T12:21:25.621457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"2dfcac63-8190-45d1-983a-dcb5bf33d348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":21348},{"type":"v1","addr":"172.21.15.172:6801","nonce":21348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":21348},{"type":"v1","addr":"172.21.15.172:6803","nonce":21348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":21348},{"type":"v1","addr":"172.21.15.172:6807","nonce":21348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":21348},{"type":"v1","addr":"172.21.15.172:6805","nonce":21348}]},"public_addr":"172.21.15.172:6801/21348","cluster_addr":"172.21.15.172:6803/21348","heartbeat_back_addr":"172.21.15.172:6807/21348","heartbeat_front_addr":"172.21.15.172:6805/21348","state":["exists","up"]},{"osd":1,"uuid":"7e338fac-a366-462f-90e5-f16f57b3706f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":21350},{"type":"v1","addr":"172.21.15.172:6809","nonce":21350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":21350},{"type":"v1","addr":"172.21.15.172:6811","nonce":21350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":21350},{"type":"v1","addr":"172.21.15.172:6816","nonce":21350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":21350},{"type":"v1","addr":"172.21.15.172:6813","nonce":21350}]},"public_addr":"172.21.15.172:6809/21350","cluster_addr":"172.21.15.172:6811/21350","heartbeat_back_addr":"172.21.15.172:6816/21350","heartbeat_front_addr":"172.21.15.172:6813/21350","state":["exists","up"]},{"osd":2,"uuid":"5987d4f0-55de-4238-adc4-ca6c9b283f18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6815","nonce":21349},{"type":"v1","addr":"172.21.15.172:6817","nonce":21349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":21349},{"type":"v1","addr":"172.21.15.172:6819","nonce":21349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":21349},{"type":"v1","addr":"172.21.15.172:6823","nonce":21349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":21349},{"type":"v1","addr":"172.21.15.172:6821","nonce":21349}]},"public_addr":"172.21.15.172:6817/21349","cluster_addr":"172.21.15.172:6819/21349","heartbeat_back_addr":"172.21.15.172:6823/21349","heartbeat_front_addr":"172.21.15.172:6821/21349","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:21:19.928750+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:21:19.906399+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:21:20.080070+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:21:28.991 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-16T12:21:28.991 INFO:tasks.ceph:Creating RBD pool 2024-09-16T12:21:28.991 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-09-16T12:21:29.718 INFO:teuthology.orchestra.run.smithi172.stderr:pool 'rbd' created 2024-09-16T12:21:29.730 DEBUG:teuthology.orchestra.run.smithi172:> rbd --cluster cluster2 pool init rbd 2024-09-16T12:21:32.740 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-09-16T12:21:32.741 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-09-16T12:21:32.741 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-09-16T12:21:32.741 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-09-16T12:21:32.742 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-16T12:21:33.078 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:33.091 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6826","nonce":24968},{"type":"v1","addr":"172.21.15.18:6827","nonce":24968}]},"active_addr":"172.21.15.18:6827/24968","active_change":"2024-09-16T12:21:17.540094+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.18:0","nonce":4032261087}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.18:0","nonce":1287853654}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.18:0","nonce":4231699877}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.18:0","nonce":1792528195}]}]} 2024-09-16T12:21:33.094 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-09-16T12:21:33.094 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-09-16T12:21:33.094 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T12:21:33.381 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:33.381 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":16,"fsid":"ba1e2d9d-316e-497e-977f-9f653ec9cac9","created":"2024-09-16T12:21:13.976545+0000","modified":"2024-09-16T12:21:32.711357+0000","last_up_change":"2024-09-16T12:21:23.671417+0000","last_in_change":"2024-09-16T12:21:14.747781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T12:21:25.621457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T12:21:29.312168+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2dfcac63-8190-45d1-983a-dcb5bf33d348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":21348},{"type":"v1","addr":"172.21.15.172:6801","nonce":21348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":21348},{"type":"v1","addr":"172.21.15.172:6803","nonce":21348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":21348},{"type":"v1","addr":"172.21.15.172:6807","nonce":21348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":21348},{"type":"v1","addr":"172.21.15.172:6805","nonce":21348}]},"public_addr":"172.21.15.172:6801/21348","cluster_addr":"172.21.15.172:6803/21348","heartbeat_back_addr":"172.21.15.172:6807/21348","heartbeat_front_addr":"172.21.15.172:6805/21348","state":["exists","up"]},{"osd":1,"uuid":"7e338fac-a366-462f-90e5-f16f57b3706f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":21350},{"type":"v1","addr":"172.21.15.172:6809","nonce":21350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":21350},{"type":"v1","addr":"172.21.15.172:6811","nonce":21350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":21350},{"type":"v1","addr":"172.21.15.172:6816","nonce":21350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":21350},{"type":"v1","addr":"172.21.15.172:6813","nonce":21350}]},"public_addr":"172.21.15.172:6809/21350","cluster_addr":"172.21.15.172:6811/21350","heartbeat_back_addr":"172.21.15.172:6816/21350","heartbeat_front_addr":"172.21.15.172:6813/21350","state":["exists","up"]},{"osd":2,"uuid":"5987d4f0-55de-4238-adc4-ca6c9b283f18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6815","nonce":21349},{"type":"v1","addr":"172.21.15.172:6817","nonce":21349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":21349},{"type":"v1","addr":"172.21.15.172:6819","nonce":21349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":21349},{"type":"v1","addr":"172.21.15.172:6823","nonce":21349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":21349},{"type":"v1","addr":"172.21.15.172:6821","nonce":21349}]},"public_addr":"172.21.15.172:6817/21349","cluster_addr":"172.21.15.172:6819/21349","heartbeat_back_addr":"172.21.15.172:6823/21349","heartbeat_front_addr":"172.21.15.172:6821/21349","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:21:19.928750+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:21:19.906399+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:21:20.080070+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:21:33.394 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-09-16T12:21:33.395 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T12:21:33.680 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:33.680 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":16,"fsid":"ba1e2d9d-316e-497e-977f-9f653ec9cac9","created":"2024-09-16T12:21:13.976545+0000","modified":"2024-09-16T12:21:32.711357+0000","last_up_change":"2024-09-16T12:21:23.671417+0000","last_in_change":"2024-09-16T12:21:14.747781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T12:21:25.621457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T12:21:29.312168+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2dfcac63-8190-45d1-983a-dcb5bf33d348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":21348},{"type":"v1","addr":"172.21.15.172:6801","nonce":21348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":21348},{"type":"v1","addr":"172.21.15.172:6803","nonce":21348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":21348},{"type":"v1","addr":"172.21.15.172:6807","nonce":21348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":21348},{"type":"v1","addr":"172.21.15.172:6805","nonce":21348}]},"public_addr":"172.21.15.172:6801/21348","cluster_addr":"172.21.15.172:6803/21348","heartbeat_back_addr":"172.21.15.172:6807/21348","heartbeat_front_addr":"172.21.15.172:6805/21348","state":["exists","up"]},{"osd":1,"uuid":"7e338fac-a366-462f-90e5-f16f57b3706f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":21350},{"type":"v1","addr":"172.21.15.172:6809","nonce":21350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":21350},{"type":"v1","addr":"172.21.15.172:6811","nonce":21350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":21350},{"type":"v1","addr":"172.21.15.172:6816","nonce":21350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":21350},{"type":"v1","addr":"172.21.15.172:6813","nonce":21350}]},"public_addr":"172.21.15.172:6809/21350","cluster_addr":"172.21.15.172:6811/21350","heartbeat_back_addr":"172.21.15.172:6816/21350","heartbeat_front_addr":"172.21.15.172:6813/21350","state":["exists","up"]},{"osd":2,"uuid":"5987d4f0-55de-4238-adc4-ca6c9b283f18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6815","nonce":21349},{"type":"v1","addr":"172.21.15.172:6817","nonce":21349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":21349},{"type":"v1","addr":"172.21.15.172:6819","nonce":21349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":21349},{"type":"v1","addr":"172.21.15.172:6823","nonce":21349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":21349},{"type":"v1","addr":"172.21.15.172:6821","nonce":21349}]},"public_addr":"172.21.15.172:6817/21349","cluster_addr":"172.21.15.172:6819/21349","heartbeat_back_addr":"172.21.15.172:6823/21349","heartbeat_front_addr":"172.21.15.172:6821/21349","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:21:19.928750+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:21:19.906399+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:21:20.080070+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T12:21:33.694 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-09-16T12:21:33.695 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-09-16T12:21:33.695 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-09-16T12:21:33.831 INFO:teuthology.orchestra.run.smithi172.stdout:34359738375 2024-09-16T12:21:33.831 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-16T12:21:33.837 INFO:teuthology.orchestra.run.smithi172.stdout:34359738374 2024-09-16T12:21:33.838 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-16T12:21:33.847 INFO:teuthology.orchestra.run.smithi172.stdout:34359738375 2024-09-16T12:21:33.847 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-16T12:21:34.129 INFO:teuthology.orchestra.run.smithi172.stdout:34359738374 2024-09-16T12:21:34.142 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738375 got 34359738374 for osd.1 2024-09-16T12:21:34.205 INFO:teuthology.orchestra.run.smithi172.stdout:34359738374 2024-09-16T12:21:34.215 INFO:teuthology.orchestra.run.smithi172.stdout:34359738373 2024-09-16T12:21:34.219 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738375 got 34359738374 for osd.2 2024-09-16T12:21:34.228 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2024-09-16T12:21:35.143 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-16T12:21:35.219 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-16T12:21:35.228 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-16T12:21:35.440 INFO:teuthology.orchestra.run.smithi172.stdout:34359738374 2024-09-16T12:21:35.458 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738375 got 34359738374 for osd.1 2024-09-16T12:21:35.499 INFO:teuthology.orchestra.run.smithi172.stdout:34359738373 2024-09-16T12:21:35.500 INFO:teuthology.orchestra.run.smithi172.stdout:34359738374 2024-09-16T12:21:35.512 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2024-09-16T12:21:35.514 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738375 got 34359738374 for osd.2 2024-09-16T12:21:36.459 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-16T12:21:36.513 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-16T12:21:36.515 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-16T12:21:36.754 INFO:teuthology.orchestra.run.smithi172.stdout:34359738376 2024-09-16T12:21:36.767 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738375 got 34359738376 for osd.1 2024-09-16T12:21:36.768 DEBUG:teuthology.parallel:result is None 2024-09-16T12:21:36.789 INFO:teuthology.orchestra.run.smithi172.stdout:34359738375 2024-09-16T12:21:36.803 INFO:teuthology.orchestra.run.smithi172.stdout:34359738376 2024-09-16T12:21:36.806 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738375 for osd.0 2024-09-16T12:21:36.806 DEBUG:teuthology.parallel:result is None 2024-09-16T12:21:36.816 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738375 got 34359738376 for osd.2 2024-09-16T12:21:36.816 DEBUG:teuthology.parallel:result is None 2024-09-16T12:21:36.817 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-16T12:21:36.817 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T12:21:37.097 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:37.100 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-09-16T12:21:37.110 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-16T12:21:35.554094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":893664,"kb_used_data":1576,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":282221856,"statfs":{"total":289910292480,"available":288995180544,"internally_reserved":0,"allocated":1613824,"data_stored":1302466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.853022"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:21:32.729862+0000","last_change":"2024-09-16T12:21:32.730202+0000","last_active":"2024-09-16T12:21:32.729862+0000","last_peered":"2024-09-16T12:21:32.729862+0000","last_clean":"2024-09-16T12:21:32.729862+0000","last_became_active":"2024-09-16T12:21:30.717557+0000","last_became_peered":"2024-09-16T12:21:30.717557+0000","last_unstale":"2024-09-16T12:21:32.729862+0000","last_undegraded":"2024-09-16T12:21:32.729862+0000","last_fullsized":"2024-09-16T12:21:32.729862+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:30.984888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087664800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:21:32.729571+0000","last_change":"2024-09-16T12:21:32.730032+0000","last_active":"2024-09-16T12:21:32.729571+0000","last_peered":"2024-09-16T12:21:32.729571+0000","last_clean":"2024-09-16T12:21:32.729571+0000","last_became_active":"2024-09-16T12:21:30.718053+0000","last_became_peered":"2024-09-16T12:21:30.718053+0000","last_unstale":"2024-09-16T12:21:32.729571+0000","last_undegraded":"2024-09-16T12:21:32.729571+0000","last_fullsized":"2024-09-16T12:21:32.729571+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:30:39.950801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000705296,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:21:32.730115+0000","last_change":"2024-09-16T12:21:32.730441+0000","last_active":"2024-09-16T12:21:32.730115+0000","last_peered":"2024-09-16T12:21:32.730115+0000","last_clean":"2024-09-16T12:21:32.730115+0000","last_became_active":"2024-09-16T12:21:30.718170+0000","last_became_peered":"2024-09-16T12:21:30.718170+0000","last_unstale":"2024-09-16T12:21:32.730115+0000","last_undegraded":"2024-09-16T12:21:32.730115+0000","last_fullsized":"2024-09-16T12:21:32.730115+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:29:35.927185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010672889999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:21:32.729459+0000","last_change":"2024-09-16T12:21:32.729794+0000","last_active":"2024-09-16T12:21:32.729459+0000","last_peered":"2024-09-16T12:21:32.729459+0000","last_clean":"2024-09-16T12:21:32.729459+0000","last_became_active":"2024-09-16T12:21:30.722079+0000","last_became_peered":"2024-09-16T12:21:30.722079+0000","last_unstale":"2024-09-16T12:21:32.729459+0000","last_undegraded":"2024-09-16T12:21:32.729459+0000","last_fullsized":"2024-09-16T12:21:32.729459+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:01:14.270858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051077700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:21:32.727153+0000","last_change":"2024-09-16T12:21:32.727153+0000","last_active":"2024-09-16T12:21:32.727153+0000","last_peered":"2024-09-16T12:21:32.727153+0000","last_clean":"2024-09-16T12:21:32.727153+0000","last_became_active":"2024-09-16T12:21:30.721610+0000","last_became_peered":"2024-09-16T12:21:30.721610+0000","last_unstale":"2024-09-16T12:21:32.727153+0000","last_undegraded":"2024-09-16T12:21:32.727153+0000","last_fullsized":"2024-09-16T12:21:32.727153+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:43:18.475061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014060990000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:21:32.719087+0000","last_change":"2024-09-16T12:21:32.719565+0000","last_active":"2024-09-16T12:21:32.719087+0000","last_peered":"2024-09-16T12:21:32.719087+0000","last_clean":"2024-09-16T12:21:32.719087+0000","last_became_active":"2024-09-16T12:21:30.724004+0000","last_became_peered":"2024-09-16T12:21:30.724004+0000","last_unstale":"2024-09-16T12:21:32.719087+0000","last_undegraded":"2024-09-16T12:21:32.719087+0000","last_fullsized":"2024-09-16T12:21:32.719087+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:05.243834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010506230000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:21:32.718906+0000","last_change":"2024-09-16T12:21:32.719558+0000","last_active":"2024-09-16T12:21:32.718906+0000","last_peered":"2024-09-16T12:21:32.718906+0000","last_clean":"2024-09-16T12:21:32.718906+0000","last_became_active":"2024-09-16T12:21:30.721621+0000","last_became_peered":"2024-09-16T12:21:30.721621+0000","last_unstale":"2024-09-16T12:21:32.718906+0000","last_undegraded":"2024-09-16T12:21:32.718906+0000","last_fullsized":"2024-09-16T12:21:32.718906+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:53:01.393711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001027781,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:21:32.730292+0000","last_change":"2024-09-16T12:21:32.730634+0000","last_active":"2024-09-16T12:21:32.730292+0000","last_peered":"2024-09-16T12:21:32.730292+0000","last_clean":"2024-09-16T12:21:32.730292+0000","last_became_active":"2024-09-16T12:21:30.717785+0000","last_became_peered":"2024-09-16T12:21:30.717785+0000","last_unstale":"2024-09-16T12:21:32.730292+0000","last_undegraded":"2024-09-16T12:21:32.730292+0000","last_fullsized":"2024-09-16T12:21:32.730292+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:30:09.189422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012262880000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'52","reported_seq":79,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:21:31.715700+0000","last_change":"2024-09-16T12:21:26.698782+0000","last_active":"2024-09-16T12:21:31.715700+0000","last_peered":"2024-09-16T12:21:31.715700+0000","last_clean":"2024-09-16T12:21:31.715700+0000","last_became_active":"2024-09-16T12:21:26.696232+0000","last_became_peered":"2024-09-16T12:21:26.696232+0000","last_unstale":"2024-09-16T12:21:31.715700+0000","last_undegraded":"2024-09-16T12:21:31.715700+0000","last_fullsized":"2024-09-16T12:21:31.715700+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:25.674706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:25.674706+0000","last_clean_scrub_stamp":"2024-09-16T12:21:25.674706+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:28.926290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738376,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":139264,"data_stored":40564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298080,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073760,"statfs":{"total":96636764160,"available":96331530240,"internally_reserved":0,"allocated":737280,"data_stored":630951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298084,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073756,"statfs":{"total":96636764160,"available":96331526144,"internally_reserved":0,"allocated":737280,"data_stored":630951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T12:21:37.112 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T12:21:37.393 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:37.396 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-09-16T12:21:37.407 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-16T12:21:35.554094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":893664,"kb_used_data":1576,"kb_used_omap":4,"kb_used_meta":892027,"kb_avail":282221856,"statfs":{"total":289910292480,"available":288995180544,"internally_reserved":0,"allocated":1613824,"data_stored":1302466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":913435656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.853022"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:21:32.729862+0000","last_change":"2024-09-16T12:21:32.730202+0000","last_active":"2024-09-16T12:21:32.729862+0000","last_peered":"2024-09-16T12:21:32.729862+0000","last_clean":"2024-09-16T12:21:32.729862+0000","last_became_active":"2024-09-16T12:21:30.717557+0000","last_became_peered":"2024-09-16T12:21:30.717557+0000","last_unstale":"2024-09-16T12:21:32.729862+0000","last_undegraded":"2024-09-16T12:21:32.729862+0000","last_fullsized":"2024-09-16T12:21:32.729862+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:30.984888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087664800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:21:32.729571+0000","last_change":"2024-09-16T12:21:32.730032+0000","last_active":"2024-09-16T12:21:32.729571+0000","last_peered":"2024-09-16T12:21:32.729571+0000","last_clean":"2024-09-16T12:21:32.729571+0000","last_became_active":"2024-09-16T12:21:30.718053+0000","last_became_peered":"2024-09-16T12:21:30.718053+0000","last_unstale":"2024-09-16T12:21:32.729571+0000","last_undegraded":"2024-09-16T12:21:32.729571+0000","last_fullsized":"2024-09-16T12:21:32.729571+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:30:39.950801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000705296,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:21:32.730115+0000","last_change":"2024-09-16T12:21:32.730441+0000","last_active":"2024-09-16T12:21:32.730115+0000","last_peered":"2024-09-16T12:21:32.730115+0000","last_clean":"2024-09-16T12:21:32.730115+0000","last_became_active":"2024-09-16T12:21:30.718170+0000","last_became_peered":"2024-09-16T12:21:30.718170+0000","last_unstale":"2024-09-16T12:21:32.730115+0000","last_undegraded":"2024-09-16T12:21:32.730115+0000","last_fullsized":"2024-09-16T12:21:32.730115+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:29:35.927185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010672889999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:21:32.729459+0000","last_change":"2024-09-16T12:21:32.729794+0000","last_active":"2024-09-16T12:21:32.729459+0000","last_peered":"2024-09-16T12:21:32.729459+0000","last_clean":"2024-09-16T12:21:32.729459+0000","last_became_active":"2024-09-16T12:21:30.722079+0000","last_became_peered":"2024-09-16T12:21:30.722079+0000","last_unstale":"2024-09-16T12:21:32.729459+0000","last_undegraded":"2024-09-16T12:21:32.729459+0000","last_fullsized":"2024-09-16T12:21:32.729459+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:01:14.270858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051077700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"16'2","reported_seq":16,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:21:32.727153+0000","last_change":"2024-09-16T12:21:32.727153+0000","last_active":"2024-09-16T12:21:32.727153+0000","last_peered":"2024-09-16T12:21:32.727153+0000","last_clean":"2024-09-16T12:21:32.727153+0000","last_became_active":"2024-09-16T12:21:30.721610+0000","last_became_peered":"2024-09-16T12:21:30.721610+0000","last_unstale":"2024-09-16T12:21:32.727153+0000","last_undegraded":"2024-09-16T12:21:32.727153+0000","last_fullsized":"2024-09-16T12:21:32.727153+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:43:18.475061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014060990000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:21:32.719087+0000","last_change":"2024-09-16T12:21:32.719565+0000","last_active":"2024-09-16T12:21:32.719087+0000","last_peered":"2024-09-16T12:21:32.719087+0000","last_clean":"2024-09-16T12:21:32.719087+0000","last_became_active":"2024-09-16T12:21:30.724004+0000","last_became_peered":"2024-09-16T12:21:30.724004+0000","last_unstale":"2024-09-16T12:21:32.719087+0000","last_undegraded":"2024-09-16T12:21:32.719087+0000","last_fullsized":"2024-09-16T12:21:32.719087+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:05.243834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010506230000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:21:32.718906+0000","last_change":"2024-09-16T12:21:32.719558+0000","last_active":"2024-09-16T12:21:32.718906+0000","last_peered":"2024-09-16T12:21:32.718906+0000","last_clean":"2024-09-16T12:21:32.718906+0000","last_became_active":"2024-09-16T12:21:30.721621+0000","last_became_peered":"2024-09-16T12:21:30.721621+0000","last_unstale":"2024-09-16T12:21:32.718906+0000","last_undegraded":"2024-09-16T12:21:32.718906+0000","last_fullsized":"2024-09-16T12:21:32.718906+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:53:01.393711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001027781,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"14'1","reported_seq":15,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-16T12:21:32.730292+0000","last_change":"2024-09-16T12:21:32.730634+0000","last_active":"2024-09-16T12:21:32.730292+0000","last_peered":"2024-09-16T12:21:32.730292+0000","last_clean":"2024-09-16T12:21:32.730292+0000","last_became_active":"2024-09-16T12:21:30.717785+0000","last_became_peered":"2024-09-16T12:21:30.717785+0000","last_unstale":"2024-09-16T12:21:32.730292+0000","last_undegraded":"2024-09-16T12:21:32.730292+0000","last_fullsized":"2024-09-16T12:21:32.730292+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:30:09.189422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012262880000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'52","reported_seq":79,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-16T12:21:31.715700+0000","last_change":"2024-09-16T12:21:26.698782+0000","last_active":"2024-09-16T12:21:31.715700+0000","last_peered":"2024-09-16T12:21:31.715700+0000","last_clean":"2024-09-16T12:21:31.715700+0000","last_became_active":"2024-09-16T12:21:26.696232+0000","last_became_peered":"2024-09-16T12:21:26.696232+0000","last_unstale":"2024-09-16T12:21:31.715700+0000","last_undegraded":"2024-09-16T12:21:31.715700+0000","last_fullsized":"2024-09-16T12:21:31.715700+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:25.674706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:25.674706+0000","last_clean_scrub_stamp":"2024-09-16T12:21:25.674706+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:28.926290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738376,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":139264,"data_stored":40564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738376,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298080,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073760,"statfs":{"total":96636764160,"available":96331530240,"internally_reserved":0,"allocated":737280,"data_stored":630951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298084,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073756,"statfs":{"total":96636764160,"available":96331526144,"internally_reserved":0,"allocated":737280,"data_stored":630951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T12:21:37.408 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-16T12:21:37.408 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-09-16T12:21:37.408 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-09-16T12:21:37.408 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-09-16T12:21:37.729 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T12:21:37.729 INFO:teuthology.orchestra.run.smithi172.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-16T12:21:37.742 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-09-16T12:21:37.742 INFO:teuthology.run_tasks:Running task exec... 2024-09-16T12:21:37.753 INFO:teuthology.task.exec:Executing custom commands... 2024-09-16T12:21:37.753 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi172.front.sepia.ceph.com 2024-09-16T12:21:37.754 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:21:38.119 INFO:teuthology.orchestra.run.smithi172.stderr:updated caps for client.mirror.4 2024-09-16T12:21:38.130 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:21:38.492 INFO:teuthology.orchestra.run.smithi172.stderr:updated caps for client.mirror.5 2024-09-16T12:21:38.503 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:21:38.874 INFO:teuthology.orchestra.run.smithi172.stderr:updated caps for client.mirror.6 2024-09-16T12:21:38.885 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.4 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:21:39.256 INFO:teuthology.orchestra.run.smithi172.stderr:updated caps for client.mirror.4 2024-09-16T12:21:39.267 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.5 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:21:39.626 INFO:teuthology.orchestra.run.smithi172.stderr:updated caps for client.mirror.5 2024-09-16T12:21:39.638 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.6 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:21:39.999 INFO:teuthology.orchestra.run.smithi172.stderr:updated caps for client.mirror.6 2024-09-16T12:21:40.010 INFO:teuthology.run_tasks:Running task exec... 2024-09-16T12:21:40.020 INFO:teuthology.task.exec:Executing custom commands... 2024-09-16T12:21:40.020 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi172.front.sepia.ceph.com 2024-09-16T12:21:40.020 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:21:40.386 INFO:teuthology.orchestra.run.smithi172.stderr:updated caps for client.mirror 2024-09-16T12:21:40.397 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:21:40.776 INFO:teuthology.orchestra.run.smithi172.stderr:updated caps for client.mirror.0 2024-09-16T12:21:40.787 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:21:41.160 INFO:teuthology.orchestra.run.smithi172.stderr:updated caps for client.mirror.1 2024-09-16T12:21:41.172 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:21:41.546 INFO:teuthology.orchestra.run.smithi172.stderr:updated caps for client.mirror.2 2024-09-16T12:21:41.558 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:21:41.923 INFO:teuthology.orchestra.run.smithi172.stderr:updated caps for client.mirror.3 2024-09-16T12:21:41.934 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:21:42.304 INFO:teuthology.orchestra.run.smithi172.stderr:updated caps for client.mirror 2024-09-16T12:21:42.317 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:21:42.691 INFO:teuthology.orchestra.run.smithi172.stderr:updated caps for client.mirror.0 2024-09-16T12:21:42.704 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:21:43.086 INFO:teuthology.orchestra.run.smithi172.stderr:updated caps for client.mirror.1 2024-09-16T12:21:43.097 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:21:43.470 INFO:teuthology.orchestra.run.smithi172.stderr:updated caps for client.mirror.2 2024-09-16T12:21:43.482 DEBUG:teuthology.orchestra.run.smithi172:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-16T12:21:43.848 INFO:teuthology.orchestra.run.smithi172.stderr:updated caps for client.mirror.3 2024-09-16T12:21:43.861 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T12:21:43.877 INFO:tasks.workunit:Pulling workunits from ref d75f23820806da622f561ee5ee7551015285993f 2024-09-16T12:21:43.877 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T12:21:43.877 DEBUG:teuthology.orchestra.run.smithi172:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-16T12:21:43.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T12:21:43.885 INFO:teuthology.orchestra.run.smithi172.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-09-16T12:21:43.885 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-16T12:21:43.934 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-16T12:21:43.935 DEBUG:teuthology.orchestra.run.smithi172:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-09-16T12:21:43.940 INFO:tasks.workunit:timeout=3h 2024-09-16T12:21:43.940 INFO:tasks.workunit:cleanup=True 2024-09-16T12:21:43.941 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout d75f23820806da622f561ee5ee7551015285993f 2024-09-16T12:21:43.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-09-16T12:22:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:Note: switching to 'd75f23820806da622f561ee5ee7551015285993f'. 2024-09-16T12:22:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2024-09-16T12:22:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T12:22:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T12:22:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T12:22:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2024-09-16T12:22:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T12:22:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T12:22:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2024-09-16T12:22:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: git switch -c 2024-09-16T12:22:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2024-09-16T12:22:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:Or undo this operation with: 2024-09-16T12:22:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2024-09-16T12:22:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: git switch - 2024-09-16T12:22:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2024-09-16T12:22:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T12:22:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2024-09-16T12:22:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:HEAD is now at d75f2382080 Merge branch 'wip-67762-quincy' of https://github.com/nbalacha/ceph into wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-16T12:22:35.721 DEBUG:teuthology.orchestra.run.smithi172:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-09-16T12:22:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-16T12:22:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-16T12:22:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-16T12:22:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-16T12:22:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-16T12:22:36.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-16T12:22:36.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-16T12:22:36.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-16T12:22:36.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-16T12:22:36.842 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-09-16T12:22:36.842 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-09-16T12:22:36.897 INFO:tasks.workunit:Running workunits matching rbd/compare_mirror_images.sh on cluster1.client.mirror... 2024-09-16T12:22:36.899 INFO:tasks.workunit:Running workunit rbd/compare_mirror_images.sh... 2024-09-16T12:22:36.899 DEBUG:teuthology.orchestra.run.smithi172:workunit test rbd/compare_mirror_images.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror RBD_DEVICE_TYPE=nbd adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_images.sh 2024-09-16T12:22:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG_PREFIX=image-primary 2024-09-16T12:22:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MIRROR_IMAGE_MODE=snapshot 2024-09-16T12:22:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MIRROR_POOL_MODE=image 2024-09-16T12:22:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT_PREFIX=test-primary 2024-09-16T12:22:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2024-09-16T12:22:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ RBD_MIRROR_INSTANCES=1 2024-09-16T12:22:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ RBD_MIRROR_MODE=snapshot 2024-09-16T12:22:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ RBD_MIRROR_USE_EXISTING_CLUSTER=1 2024-09-16T12:22:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_images.sh 2024-09-16T12:22:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-09-16T12:22:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ type xmlstarlet 2024-09-16T12:22:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ XMLSTARLET=xmlstarlet 2024-09-16T12:22:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ RBD_MIRROR_INSTANCES=1 2024-09-16T12:22:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CLUSTER1=cluster1 2024-09-16T12:22:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CLUSTER2=cluster2 2024-09-16T12:22:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ PEER_CLUSTER_SUFFIX= 2024-09-16T12:22:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ POOL=mirror 2024-09-16T12:22:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ PARENT_POOL=mirror_parent 2024-09-16T12:22:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ NS1=ns1 2024-09-16T12:22:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ NS2=ns2 2024-09-16T12:22:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ TEMPDIR= 2024-09-16T12:22:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ID=mirror 2024-09-16T12:22:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock,object-map,fast-diff 2024-09-16T12:22:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-09-16T12:22:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ MIRROR_POOL_MODE=image 2024-09-16T12:22:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ MIRROR_IMAGE_MODE=snapshot 2024-09-16T12:22:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-09-16T12:22:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS='--id mirror' 2024-09-16T12:22:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ LAST_MIRROR_INSTANCE=0 2024-09-16T12:22:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/compare_mirror_images.sh 2024-09-16T12:22:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-09-16T12:22:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-09-16T12:22:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_BIN=. 2024-09-16T12:22:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_SRC=. 2024-09-16T12:22:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ '[' -e CMakeCache.txt ']' 2024-09-16T12:22:36.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ '[' 0 -gt 0 ']' 2024-09-16T12:22:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ setup 2024-09-16T12:22:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local c 2024-09-16T12:22:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-09-16T12:22:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ setup_tempdir 2024-09-16T12:22:36.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:22:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ mktemp -d 2024-09-16T12:22:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ TEMPDIR=/tmp/tmp.VxztfXQF7C 2024-09-16T12:22:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -z 1 ']' 2024-09-16T12:22:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ setup_pools cluster1 cluster2 2024-09-16T12:22:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:22:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local remote_cluster=cluster2 2024-09-16T12:22:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mon_map_file 2024-09-16T12:22:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mon_addr 2024-09-16T12:22:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local admin_key_file 2024-09-16T12:22:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local uuid 2024-09-16T12:22:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ CEPH_ARGS= 2024-09-16T12:22:36.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-09-16T12:22:38.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:pool 'mirror' already exists 2024-09-16T12:22:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ CEPH_ARGS= 2024-09-16T12:22:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-09-16T12:22:39.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:pool 'mirror_parent' already exists 2024-09-16T12:22:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ CEPH_ARGS= 2024-09-16T12:22:39.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 pool init mirror 2024-09-16T12:22:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ CEPH_ARGS= 2024-09-16T12:22:42.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-09-16T12:22:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:22:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ CEPH_ARGS= 2024-09-16T12:22:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-09-16T12:22:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-09-16T12:22:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-09-16T12:22:45.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-09-16T12:22:45.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-09-16T12:22:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-09-16T12:22:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -z ']' 2024-09-16T12:22:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -z ']' 2024-09-16T12:22:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ peer_add cluster1 mirror cluster2 2024-09-16T12:22:45.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:22:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:22:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:22:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:22:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local client_cluster=cluster2 2024-09-16T12:22:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:22:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local remote_cluster=cluster2 2024-09-16T12:22:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local uuid_var_name 2024-09-16T12:22:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:22:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local error_code 2024-09-16T12:22:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local peer_uuid 2024-09-16T12:22:45.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T12:22:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ set +e 2024-09-16T12:22:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-09-16T12:22:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ peer_uuid=322491e7-7901-4e96-bc59-78bf886dde68 2024-09-16T12:22:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ error_code=0 2024-09-16T12:22:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ set -e 2024-09-16T12:22:45.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T12:22:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test 0 -eq 0 2024-09-16T12:22:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:22:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:22:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-09-16T12:22:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:22:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:22:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror_parent 2024-09-16T12:22:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:22:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local client_cluster=cluster2 2024-09-16T12:22:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:22:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local remote_cluster=cluster2 2024-09-16T12:22:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local uuid_var_name 2024-09-16T12:22:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:22:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local error_code 2024-09-16T12:22:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local peer_uuid 2024-09-16T12:22:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T12:22:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ set +e 2024-09-16T12:22:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-09-16T12:22:45.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ peer_uuid=4115c9ab-2e63-4d11-9694-277c427daa8e 2024-09-16T12:22:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ error_code=0 2024-09-16T12:22:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ set -e 2024-09-16T12:22:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T12:22:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test 0 -eq 0 2024-09-16T12:22:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:22:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:22:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ setup_pools cluster2 cluster1 2024-09-16T12:22:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:22:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local remote_cluster=cluster1 2024-09-16T12:22:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mon_map_file 2024-09-16T12:22:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mon_addr 2024-09-16T12:22:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local admin_key_file 2024-09-16T12:22:45.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local uuid 2024-09-16T12:22:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ CEPH_ARGS= 2024-09-16T12:22:45.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-09-16T12:22:46.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:pool 'mirror' already exists 2024-09-16T12:22:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ CEPH_ARGS= 2024-09-16T12:22:46.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-09-16T12:22:47.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:pool 'mirror_parent' already exists 2024-09-16T12:22:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ CEPH_ARGS= 2024-09-16T12:22:47.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster2 pool init mirror 2024-09-16T12:22:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ CEPH_ARGS= 2024-09-16T12:22:50.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-09-16T12:22:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:22:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ CEPH_ARGS= 2024-09-16T12:22:53.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-09-16T12:22:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-09-16T12:22:53.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-09-16T12:22:53.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-09-16T12:22:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-09-16T12:22:53.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-09-16T12:22:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -z ']' 2024-09-16T12:22:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -z ']' 2024-09-16T12:22:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ peer_add cluster2 mirror cluster1 2024-09-16T12:22:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:22:53.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:22:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:22:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:22:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local client_cluster=cluster1 2024-09-16T12:22:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:22:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local remote_cluster=cluster1 2024-09-16T12:22:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local uuid_var_name 2024-09-16T12:22:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:22:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local error_code 2024-09-16T12:22:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local peer_uuid 2024-09-16T12:22:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T12:22:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ set +e 2024-09-16T12:22:53.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-09-16T12:22:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ peer_uuid=f622e86b-f447-4bd5-90ed-dfe9919f31af 2024-09-16T12:22:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ error_code=0 2024-09-16T12:22:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ set -e 2024-09-16T12:22:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T12:22:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test 0 -eq 0 2024-09-16T12:22:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:22:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:22:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-09-16T12:22:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:22:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:22:53.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror_parent 2024-09-16T12:22:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:22:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local client_cluster=cluster1 2024-09-16T12:22:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:22:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local remote_cluster=cluster1 2024-09-16T12:22:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local uuid_var_name 2024-09-16T12:22:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:22:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local error_code 2024-09-16T12:22:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local peer_uuid 2024-09-16T12:22:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T12:22:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ set +e 2024-09-16T12:22:53.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-09-16T12:22:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ peer_uuid=b0b624ab-2fc5-4906-9e44-d27e80d47be6 2024-09-16T12:22:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ error_code=0 2024-09-16T12:22:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ set -e 2024-09-16T12:22:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' 0 -eq 17 ']' 2024-09-16T12:22:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test 0 -eq 0 2024-09-16T12:22:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:22:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:22:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:22:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ start_mirrors cluster1 2024-09-16T12:22:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:22:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ seq 0 0 2024-09-16T12:22:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:22:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ start_mirror cluster1:0 2024-09-16T12:22:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1:0 2024-09-16T12:22:53.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local instance 2024-09-16T12:22:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-16T12:22:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local val=cluster1:0 2024-09-16T12:22:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster_var_name=cluster 2024-09-16T12:22:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local instance_var_name=instance 2024-09-16T12:22:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cluster=cluster1 2024-09-16T12:22:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ instance=0 2024-09-16T12:22:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-16T12:22:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ eval cluster=cluster1 2024-09-16T12:22:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ cluster=cluster1 2024-09-16T12:22:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ eval instance=0 2024-09-16T12:22:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ instance=0 2024-09-16T12:22:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test -n '' 2024-09-16T12:22:53.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd-mirror --cluster cluster1 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-16T12:22:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ start_mirrors cluster2 2024-09-16T12:22:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:22:53.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ seq 0 0 2024-09-16T12:22:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T12:22:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ start_mirror cluster2:0 2024-09-16T12:22:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2:0 2024-09-16T12:22:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local instance 2024-09-16T12:22:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-16T12:22:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local val=cluster2:0 2024-09-16T12:22:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster_var_name=cluster 2024-09-16T12:22:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local instance_var_name=instance 2024-09-16T12:22:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cluster=cluster2 2024-09-16T12:22:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ instance=0 2024-09-16T12:22:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-16T12:22:53.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ eval cluster=cluster2 2024-09-16T12:22:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ cluster=cluster2 2024-09-16T12:22:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ eval instance=0 2024-09-16T12:22:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ instance=0 2024-09-16T12:22:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test -n '' 2024-09-16T12:22:53.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd-mirror --cluster cluster2 --id mirror.0 --rbd-mirror-delete-retry-interval=5 --rbd-mirror-image-state-check-interval=5 --rbd-mirror-journal-poll-age=1 --rbd-mirror-pool-replayers-refresh-interval=5 --debug-rbd=30 --debug-journaler=30 --debug-rbd_mirror=30 --daemonize=true 2024-09-16T12:22:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wget https://download.ceph.com/qa/linux-5.4.tar.gz 2024-09-16T12:22:53.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:--2024-09-16 12:22:53-- https://download.ceph.com/qa/linux-5.4.tar.gz 2024-09-16T12:22:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-09-16T12:22:53.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:443... connected. 2024-09-16T12:22:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:HTTP request sent, awaiting response... 200 OK 2024-09-16T12:22:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:Length: 172616875 (165M) [application/octet-stream] 2024-09-16T12:22:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:Saving to: 'linux-5.4.tar.gz' 2024-09-16T12:22:53.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2024-09-16T12:22:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 0K .......... .......... .......... .......... .......... 0% 2.40M 68s 2024-09-16T12:22:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50K .......... .......... .......... .......... .......... 0% 2.38M 69s 2024-09-16T12:22:53.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 100K .......... .......... .......... .......... .......... 0% 146M 46s 2024-09-16T12:22:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 150K .......... .......... .......... .......... .......... 0% 2.46M 51s 2024-09-16T12:22:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 200K .......... .......... .......... .......... .......... 0% 151M 41s 2024-09-16T12:22:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 250K .......... .......... .......... .......... .......... 0% 193M 35s 2024-09-16T12:22:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 300K .......... .......... .......... .......... .......... 0% 192M 30s 2024-09-16T12:22:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 350K .......... .......... .......... .......... .......... 0% 151M 26s 2024-09-16T12:22:53.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 400K .......... .......... .......... .......... .......... 0% 2.52M 30s 2024-09-16T12:22:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 450K .......... .......... .......... .......... .......... 0% 90.2M 28s 2024-09-16T12:22:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 500K .......... .......... .......... .......... .......... 0% 140M 25s 2024-09-16T12:22:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 550K .......... .......... .......... .......... .......... 0% 118M 23s 2024-09-16T12:22:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 600K .......... .......... .......... .......... .......... 0% 110M 22s 2024-09-16T12:22:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 650K .......... .......... .......... .......... .......... 0% 86.2M 20s 2024-09-16T12:22:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 700K .......... .......... .......... .......... .......... 0% 91.9M 19s 2024-09-16T12:22:53.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 750K .......... .......... .......... .......... .......... 0% 49.2M 18s 2024-09-16T12:22:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 800K .......... .......... .......... .......... .......... 0% 3.03M 20s 2024-09-16T12:22:53.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 850K .......... .......... .......... .......... .......... 0% 131M 19s 2024-09-16T12:22:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 900K .......... .......... .......... .......... .......... 0% 134M 18s 2024-09-16T12:22:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 950K .......... .......... .......... .......... .......... 0% 158M 17s 2024-09-16T12:22:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 1000K .......... .......... .......... .......... .......... 0% 217M 16s 2024-09-16T12:22:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 1050K .......... .......... .......... .......... .......... 0% 230M 16s 2024-09-16T12:22:53.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 1100K .......... .......... .......... .......... .......... 0% 205M 15s 2024-09-16T12:22:53.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 1150K .......... .......... .......... .......... .......... 0% 50.5M 15s 2024-09-16T12:22:53.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 1200K .......... .......... .......... .......... .......... 0% 34.5M 14s 2024-09-16T12:22:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 1250K .......... .......... .......... .......... .......... 0% 22.7M 14s 2024-09-16T12:22:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 1300K .......... .......... .......... .......... .......... 0% 25.0M 14s 2024-09-16T12:22:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 1350K .......... .......... .......... .......... .......... 0% 55.5M 13s 2024-09-16T12:22:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 1400K .......... .......... .......... .......... .......... 0% 31.5M 13s 2024-09-16T12:22:53.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 1450K .......... .......... .......... .......... .......... 0% 24.5M 13s 2024-09-16T12:22:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 1500K .......... .......... .......... .......... .......... 0% 23.3M 13s 2024-09-16T12:22:53.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 1550K .......... .......... .......... .......... .......... 0% 51.9M 12s 2024-09-16T12:22:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 1600K .......... .......... .......... .......... .......... 0% 27.0M 12s 2024-09-16T12:22:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 1650K .......... .......... .......... .......... .......... 1% 29.6M 12s 2024-09-16T12:22:53.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 1700K .......... .......... .......... .......... .......... 1% 23.5M 12s 2024-09-16T12:22:53.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 1750K .......... .......... .......... .......... .......... 1% 22.0M 12s 2024-09-16T12:22:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 1800K .......... .......... .......... .......... .......... 1% 26.2M 11s 2024-09-16T12:22:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 1850K .......... .......... .......... .......... .......... 1% 114M 11s 2024-09-16T12:22:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 1900K .......... .......... .......... .......... .......... 1% 24.3M 11s 2024-09-16T12:22:53.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 1950K .......... .......... .......... .......... .......... 1% 23.5M 11s 2024-09-16T12:22:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2000K .......... .......... .......... .......... .......... 1% 22.9M 11s 2024-09-16T12:22:53.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2050K .......... .......... .......... .......... .......... 1% 50.1M 11s 2024-09-16T12:22:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2100K .......... .......... .......... .......... .......... 1% 36.0M 11s 2024-09-16T12:22:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2150K .......... .......... .......... .......... .......... 1% 24.0M 10s 2024-09-16T12:22:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2200K .......... .......... .......... .......... .......... 1% 40.0M 10s 2024-09-16T12:22:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2250K .......... .......... .......... .......... .......... 1% 25.6M 10s 2024-09-16T12:22:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2300K .......... .......... .......... .......... .......... 1% 23.5M 10s 2024-09-16T12:22:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2350K .......... .......... .......... .......... .......... 1% 22.0M 10s 2024-09-16T12:22:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2400K .......... .......... .......... .......... .......... 1% 57.2M 10s 2024-09-16T12:22:53.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2450K .......... .......... .......... .......... .......... 1% 23.8M 10s 2024-09-16T12:22:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2500K .......... .......... .......... .......... .......... 1% 35.5M 10s 2024-09-16T12:22:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2550K .......... .......... .......... .......... .......... 1% 25.3M 10s 2024-09-16T12:22:53.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2600K .......... .......... .......... .......... .......... 1% 44.1M 10s 2024-09-16T12:22:53.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2650K .......... .......... .......... .......... .......... 1% 28.7M 9s 2024-09-16T12:22:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2700K .......... .......... .......... .......... .......... 1% 26.1M 9s 2024-09-16T12:22:53.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2750K .......... .......... .......... .......... .......... 1% 23.4M 9s 2024-09-16T12:22:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2800K .......... .......... .......... .......... .......... 1% 24.8M 9s 2024-09-16T12:22:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2850K .......... .......... .......... .......... .......... 1% 23.5M 9s 2024-09-16T12:22:53.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2900K .......... .......... .......... .......... .......... 1% 83.0M 9s 2024-09-16T12:22:53.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2950K .......... .......... .......... .......... .......... 1% 31.7M 9s 2024-09-16T12:22:53.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 3000K .......... .......... .......... .......... .......... 1% 23.4M 9s 2024-09-16T12:22:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 3050K .......... .......... .......... .......... .......... 1% 41.3M 9s 2024-09-16T12:22:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 3100K .......... .......... .......... .......... .......... 1% 28.7M 9s 2024-09-16T12:22:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 3150K .......... .......... .......... .......... .......... 1% 27.6M 9s 2024-09-16T12:22:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 3200K .......... .......... .......... .......... .......... 1% 29.2M 9s 2024-09-16T12:22:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 3250K .......... .......... .......... .......... .......... 1% 19.6M 9s 2024-09-16T12:22:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 3300K .......... .......... .......... .......... .......... 1% 36.0M 9s 2024-09-16T12:22:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 3350K .......... .......... .......... .......... .......... 2% 44.6M 9s 2024-09-16T12:22:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 3400K .......... .......... .......... .......... .......... 2% 25.4M 9s 2024-09-16T12:22:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 3450K .......... .......... .......... .......... .......... 2% 32.4M 9s 2024-09-16T12:22:53.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 3500K .......... .......... .......... .......... .......... 2% 24.9M 9s 2024-09-16T12:22:53.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 3550K .......... .......... .......... .......... .......... 2% 25.8M 8s 2024-09-16T12:22:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 3600K .......... .......... .......... .......... .......... 2% 22.7M 8s 2024-09-16T12:22:53.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 3650K .......... .......... .......... .......... .......... 2% 49.0M 8s 2024-09-16T12:22:53.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 3700K .......... .......... .......... .......... .......... 2% 36.5M 8s 2024-09-16T12:22:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 3750K .......... .......... .......... .......... .......... 2% 24.9M 8s 2024-09-16T12:22:53.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 3800K .......... .......... .......... .......... .......... 2% 24.2M 8s 2024-09-16T12:22:53.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 3850K .......... .......... .......... .......... .......... 2% 49.3M 8s 2024-09-16T12:22:53.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 3900K .......... .......... .......... .......... .......... 2% 26.3M 8s 2024-09-16T12:22:53.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 3950K .......... .......... .......... .......... .......... 2% 22.9M 8s 2024-09-16T12:22:53.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 4000K .......... .......... .......... .......... .......... 2% 36.2M 8s 2024-09-16T12:22:53.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 4050K .......... .......... .......... .......... .......... 2% 23.0M 8s 2024-09-16T12:22:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 4100K .......... .......... .......... .......... .......... 2% 25.2M 8s 2024-09-16T12:22:53.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 4150K .......... .......... .......... .......... .......... 2% 52.8M 8s 2024-09-16T12:22:53.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 4200K .......... .......... .......... .......... .......... 2% 23.3M 8s 2024-09-16T12:22:53.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 4250K .......... .......... .......... .......... .......... 2% 34.9M 8s 2024-09-16T12:22:53.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 4300K .......... .......... .......... .......... .......... 2% 21.8M 8s 2024-09-16T12:22:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 4350K .......... .......... .......... .......... .......... 2% 30.9M 8s 2024-09-16T12:22:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 4400K .......... .......... .......... .......... .......... 2% 44.2M 8s 2024-09-16T12:22:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 4450K .......... .......... .......... .......... .......... 2% 25.2M 8s 2024-09-16T12:22:53.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 4500K .......... .......... .......... .......... .......... 2% 23.6M 8s 2024-09-16T12:22:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 4550K .......... .......... .......... .......... .......... 2% 31.1M 8s 2024-09-16T12:22:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 4600K .......... .......... .......... .......... .......... 2% 26.9M 8s 2024-09-16T12:22:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 4650K .......... .......... .......... .......... .......... 2% 56.6M 8s 2024-09-16T12:22:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 4700K .......... .......... .......... .......... .......... 2% 33.5M 8s 2024-09-16T12:22:53.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 4750K .......... .......... .......... .......... .......... 2% 18.3M 8s 2024-09-16T12:22:53.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 4800K .......... .......... .......... .......... .......... 2% 38.3M 8s 2024-09-16T12:22:53.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 4850K .......... .......... .......... .......... .......... 2% 24.4M 8s 2024-09-16T12:22:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 4900K .......... .......... .......... .......... .......... 2% 49.3M 8s 2024-09-16T12:22:53.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 4950K .......... .......... .......... .......... .......... 2% 24.6M 8s 2024-09-16T12:22:53.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 5000K .......... .......... .......... .......... .......... 2% 35.3M 8s 2024-09-16T12:22:53.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 5050K .......... .......... .......... .......... .......... 3% 25.0M 8s 2024-09-16T12:22:53.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 5100K .......... .......... .......... .......... .......... 3% 51.8M 7s 2024-09-16T12:22:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 5150K .......... .......... .......... .......... .......... 3% 15.4M 8s 2024-09-16T12:22:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 5200K .......... .......... .......... .......... .......... 3% 55.9M 7s 2024-09-16T12:22:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 5250K .......... .......... .......... .......... .......... 3% 35.2M 7s 2024-09-16T12:22:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 5300K .......... .......... .......... .......... .......... 3% 23.6M 7s 2024-09-16T12:22:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 5350K .......... .......... .......... .......... .......... 3% 24.6M 7s 2024-09-16T12:22:53.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 5400K .......... .......... .......... .......... .......... 3% 52.5M 7s 2024-09-16T12:22:53.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 5450K .......... .......... .......... .......... .......... 3% 24.1M 7s 2024-09-16T12:22:53.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 5500K .......... .......... .......... .......... .......... 3% 25.3M 7s 2024-09-16T12:22:53.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 5550K .......... .......... .......... .......... .......... 3% 34.1M 7s 2024-09-16T12:22:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 5600K .......... .......... .......... .......... .......... 3% 21.1M 7s 2024-09-16T12:22:53.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 5650K .......... .......... .......... .......... .......... 3% 94.1M 7s 2024-09-16T12:22:53.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 5700K .......... .......... .......... .......... .......... 3% 22.9M 7s 2024-09-16T12:22:53.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 5750K .......... .......... .......... .......... .......... 3% 27.1M 7s 2024-09-16T12:22:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 5800K .......... .......... .......... .......... .......... 3% 31.9M 7s 2024-09-16T12:22:53.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 5850K .......... .......... .......... .......... .......... 3% 23.9M 7s 2024-09-16T12:22:53.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 5900K .......... .......... .......... .......... .......... 3% 47.3M 7s 2024-09-16T12:22:53.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 5950K .......... .......... .......... .......... .......... 3% 25.4M 7s 2024-09-16T12:22:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 6000K .......... .......... .......... .......... .......... 3% 23.3M 7s 2024-09-16T12:22:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 6050K .......... .......... .......... .......... .......... 3% 32.8M 7s 2024-09-16T12:22:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 6100K .......... .......... .......... .......... .......... 3% 28.1M 7s 2024-09-16T12:22:53.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 6150K .......... .......... .......... .......... .......... 3% 42.5M 7s 2024-09-16T12:22:53.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 6200K .......... .......... .......... .......... .......... 3% 24.3M 7s 2024-09-16T12:22:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 6250K .......... .......... .......... .......... .......... 3% 34.3M 7s 2024-09-16T12:22:53.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 6300K .......... .......... .......... .......... .......... 3% 22.4M 7s 2024-09-16T12:22:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 6350K .......... .......... .......... .......... .......... 3% 10.9M 7s 2024-09-16T12:22:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 6400K .......... .......... .......... .......... .......... 3% 115M 7s 2024-09-16T12:22:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 6450K .......... .......... .......... .......... .......... 3% 157M 7s 2024-09-16T12:22:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 6500K .......... .......... .......... .......... .......... 3% 39.7M 7s 2024-09-16T12:22:53.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 6550K .......... .......... .......... .......... .......... 3% 27.5M 7s 2024-09-16T12:22:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 6600K .......... .......... .......... .......... .......... 3% 42.7M 7s 2024-09-16T12:22:53.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 6650K .......... .......... .......... .......... .......... 3% 29.6M 7s 2024-09-16T12:22:53.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 6700K .......... .......... .......... .......... .......... 4% 20.0M 7s 2024-09-16T12:22:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 6750K .......... .......... .......... .......... .......... 4% 22.6M 7s 2024-09-16T12:22:53.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 6800K .......... .......... .......... .......... .......... 4% 25.6M 7s 2024-09-16T12:22:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 6850K .......... .......... .......... .......... .......... 4% 47.1M 7s 2024-09-16T12:22:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 6900K .......... .......... .......... .......... .......... 4% 31.3M 7s 2024-09-16T12:22:53.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 6950K .......... .......... .......... .......... .......... 4% 28.2M 7s 2024-09-16T12:22:53.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 7000K .......... .......... .......... .......... .......... 4% 18.3M 7s 2024-09-16T12:22:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 7050K .......... .......... .......... .......... .......... 4% 37.2M 7s 2024-09-16T12:22:53.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 7100K .......... .......... .......... .......... .......... 4% 23.0M 7s 2024-09-16T12:22:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 7150K .......... .......... .......... .......... .......... 4% 25.7M 7s 2024-09-16T12:22:53.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 7200K .......... .......... .......... .......... .......... 4% 48.5M 7s 2024-09-16T12:22:53.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 7250K .......... .......... .......... .......... .......... 4% 23.6M 7s 2024-09-16T12:22:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 7300K .......... .......... .......... .......... .......... 4% 35.6M 7s 2024-09-16T12:22:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 7350K .......... .......... .......... .......... .......... 4% 23.8M 7s 2024-09-16T12:22:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 7400K .......... .......... .......... .......... .......... 4% 30.0M 7s 2024-09-16T12:22:53.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 7450K .......... .......... .......... .......... .......... 4% 36.9M 7s 2024-09-16T12:22:53.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 7500K .......... .......... .......... .......... .......... 4% 32.1M 7s 2024-09-16T12:22:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 7550K .......... .......... .......... .......... .......... 4% 22.9M 7s 2024-09-16T12:22:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 7600K .......... .......... .......... .......... .......... 4% 25.2M 7s 2024-09-16T12:22:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 7650K .......... .......... .......... .......... .......... 4% 25.2M 7s 2024-09-16T12:22:54.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 7700K .......... .......... .......... .......... .......... 4% 49.9M 7s 2024-09-16T12:22:54.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 7750K .......... .......... .......... .......... .......... 4% 25.1M 7s 2024-09-16T12:22:54.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 7800K .......... .......... .......... .......... .......... 4% 33.4M 7s 2024-09-16T12:22:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 7850K .......... .......... .......... .......... .......... 4% 27.1M 7s 2024-09-16T12:22:54.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 7900K .......... .......... .......... .......... .......... 4% 52.8M 7s 2024-09-16T12:22:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 7950K .......... .......... .......... .......... .......... 4% 17.4M 7s 2024-09-16T12:22:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 8000K .......... .......... .......... .......... .......... 4% 35.4M 7s 2024-09-16T12:22:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 8050K .......... .......... .......... .......... .......... 4% 30.2M 7s 2024-09-16T12:22:54.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 8100K .......... .......... .......... .......... .......... 4% 20.8M 7s 2024-09-16T12:22:54.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 8150K .......... .......... .......... .......... .......... 4% 32.7M 7s 2024-09-16T12:22:54.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 8200K .......... .......... .......... .......... .......... 4% 47.4M 7s 2024-09-16T12:22:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 8250K .......... .......... .......... .......... .......... 4% 31.7M 7s 2024-09-16T12:22:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 8300K .......... .......... .......... .......... .......... 4% 27.5M 7s 2024-09-16T12:22:54.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 8350K .......... .......... .......... .......... .......... 4% 19.6M 7s 2024-09-16T12:22:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 8400K .......... .......... .......... .......... .......... 5% 33.6M 7s 2024-09-16T12:22:54.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 8450K .......... .......... .......... .......... .......... 5% 45.6M 7s 2024-09-16T12:22:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 8500K .......... .......... .......... .......... .......... 5% 25.3M 7s 2024-09-16T12:22:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 8550K .......... .......... .......... .......... .......... 5% 34.4M 7s 2024-09-16T12:22:54.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 8600K .......... .......... .......... .......... .......... 5% 23.7M 7s 2024-09-16T12:22:54.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 8650K .......... .......... .......... .......... .......... 5% 24.1M 7s 2024-09-16T12:22:54.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 8700K .......... .......... .......... .......... .......... 5% 57.6M 7s 2024-09-16T12:22:54.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 8750K .......... .......... .......... .......... .......... 5% 23.0M 7s 2024-09-16T12:22:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 8800K .......... .......... .......... .......... .......... 5% 24.8M 7s 2024-09-16T12:22:54.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 8850K .......... .......... .......... .......... .......... 5% 34.7M 7s 2024-09-16T12:22:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 8900K .......... .......... .......... .......... .......... 5% 25.6M 7s 2024-09-16T12:22:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 8950K .......... .......... .......... .......... .......... 5% 51.6M 7s 2024-09-16T12:22:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 9000K .......... .......... .......... .......... .......... 5% 30.8M 6s 2024-09-16T12:22:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 9050K .......... .......... .......... .......... .......... 5% 25.2M 6s 2024-09-16T12:22:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 9100K .......... .......... .......... .......... .......... 5% 24.3M 6s 2024-09-16T12:22:54.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 9150K .......... .......... .......... .......... .......... 5% 23.6M 6s 2024-09-16T12:22:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 9200K .......... .......... .......... .......... .......... 5% 53.4M 6s 2024-09-16T12:22:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 9250K .......... .......... .......... .......... .......... 5% 26.6M 6s 2024-09-16T12:22:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 9300K .......... .......... .......... .......... .......... 5% 22.8M 6s 2024-09-16T12:22:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 9350K .......... .......... .......... .......... .......... 5% 33.8M 6s 2024-09-16T12:22:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 9400K .......... .......... .......... .......... .......... 5% 24.0M 6s 2024-09-16T12:22:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 9450K .......... .......... .......... .......... .......... 5% 51.3M 6s 2024-09-16T12:22:54.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 9500K .......... .......... .......... .......... .......... 5% 25.0M 6s 2024-09-16T12:22:54.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 9550K .......... .......... .......... .......... .......... 5% 24.6M 6s 2024-09-16T12:22:54.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 9600K .......... .......... .......... .......... .......... 5% 22.7M 6s 2024-09-16T12:22:54.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 9650K .......... .......... .......... .......... .......... 5% 32.9M 6s 2024-09-16T12:22:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 9700K .......... .......... .......... .......... .......... 5% 27.1M 6s 2024-09-16T12:22:54.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 9750K .......... .......... .......... .......... .......... 5% 51.5M 6s 2024-09-16T12:22:54.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 9800K .......... .......... .......... .......... .......... 5% 32.6M 6s 2024-09-16T12:22:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 9850K .......... .......... .......... .......... .......... 5% 19.5M 6s 2024-09-16T12:22:54.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 9900K .......... .......... .......... .......... .......... 5% 44.1M 6s 2024-09-16T12:22:54.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 9950K .......... .......... .......... .......... .......... 5% 17.5M 6s 2024-09-16T12:22:54.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 10000K .......... .......... .......... .......... .......... 5% 49.3M 6s 2024-09-16T12:22:54.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 10050K .......... .......... .......... .......... .......... 5% 25.0M 6s 2024-09-16T12:22:54.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 10100K .......... .......... .......... .......... .......... 6% 27.1M 6s 2024-09-16T12:22:54.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 10150K .......... .......... .......... .......... .......... 6% 25.3M 6s 2024-09-16T12:22:54.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 10200K .......... .......... .......... .......... .......... 6% 45.0M 6s 2024-09-16T12:22:54.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 10250K .......... .......... .......... .......... .......... 6% 24.0M 6s 2024-09-16T12:22:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 10300K .......... .......... .......... .......... .......... 6% 32.2M 6s 2024-09-16T12:22:54.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 10350K .......... .......... .......... .......... .......... 6% 24.1M 6s 2024-09-16T12:22:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 10400K .......... .......... .......... .......... .......... 6% 24.7M 6s 2024-09-16T12:22:54.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 10450K .......... .......... .......... .......... .......... 6% 25.3M 6s 2024-09-16T12:22:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 10500K .......... .......... .......... .......... .......... 6% 48.8M 6s 2024-09-16T12:22:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 10550K .......... .......... .......... .......... .......... 6% 24.3M 6s 2024-09-16T12:22:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 10600K .......... .......... .......... .......... .......... 6% 38.2M 6s 2024-09-16T12:22:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 10650K .......... .......... .......... .......... .......... 6% 23.7M 6s 2024-09-16T12:22:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 10700K .......... .......... .......... .......... .......... 6% 54.7M 6s 2024-09-16T12:22:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 10750K .......... .......... .......... .......... .......... 6% 23.2M 6s 2024-09-16T12:22:54.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 10800K .......... .......... .......... .......... .......... 6% 25.6M 6s 2024-09-16T12:22:54.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 10850K .......... .......... .......... .......... .......... 6% 31.9M 6s 2024-09-16T12:22:54.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 10900K .......... .......... .......... .......... .......... 6% 23.2M 6s 2024-09-16T12:22:54.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 10950K .......... .......... .......... .......... .......... 6% 32.4M 6s 2024-09-16T12:22:54.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 11000K .......... .......... .......... .......... .......... 6% 35.3M 6s 2024-09-16T12:22:54.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 11050K .......... .......... .......... .......... .......... 6% 24.1M 6s 2024-09-16T12:22:54.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 11100K .......... .......... .......... .......... .......... 6% 35.6M 6s 2024-09-16T12:22:54.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 11150K .......... .......... .......... .......... .......... 6% 22.6M 6s 2024-09-16T12:22:54.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 11200K .......... .......... .......... .......... .......... 6% 25.8M 6s 2024-09-16T12:22:54.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 11250K .......... .......... .......... .......... .......... 6% 43.4M 6s 2024-09-16T12:22:54.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 11300K .......... .......... .......... .......... .......... 6% 27.3M 6s 2024-09-16T12:22:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 11350K .......... .......... .......... .......... .......... 6% 32.7M 6s 2024-09-16T12:22:54.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 11400K .......... .......... .......... .......... .......... 6% 26.3M 6s 2024-09-16T12:22:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 11450K .......... .......... .......... .......... .......... 6% 21.7M 6s 2024-09-16T12:22:54.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 11500K .......... .......... .......... .......... .......... 6% 55.7M 6s 2024-09-16T12:22:54.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 11550K .......... .......... .......... .......... .......... 6% 22.8M 6s 2024-09-16T12:22:54.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 11600K .......... .......... .......... .......... .......... 6% 25.3M 6s 2024-09-16T12:22:54.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 11650K .......... .......... .......... .......... .......... 6% 36.9M 6s 2024-09-16T12:22:54.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 11700K .......... .......... .......... .......... .......... 6% 24.7M 6s 2024-09-16T12:22:54.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 11750K .......... .......... .......... .......... .......... 7% 43.2M 6s 2024-09-16T12:22:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 11800K .......... .......... .......... .......... .......... 7% 26.3M 6s 2024-09-16T12:22:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 11850K .......... .......... .......... .......... .......... 7% 36.4M 6s 2024-09-16T12:22:54.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 11900K .......... .......... .......... .......... .......... 7% 24.4M 6s 2024-09-16T12:22:54.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 11950K .......... .......... .......... .......... .......... 7% 22.7M 6s 2024-09-16T12:22:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 12000K .......... .......... .......... .......... .......... 7% 24.7M 6s 2024-09-16T12:22:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 12050K .......... .......... .......... .......... .......... 7% 53.9M 6s 2024-09-16T12:22:54.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 12100K .......... .......... .......... .......... .......... 7% 33.7M 6s 2024-09-16T12:22:54.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 12150K .......... .......... .......... .......... .......... 7% 28.6M 6s 2024-09-16T12:22:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 12200K .......... .......... .......... .......... .......... 7% 20.8M 6s 2024-09-16T12:22:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 12250K .......... .......... .......... .......... .......... 7% 60.1M 6s 2024-09-16T12:22:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 12300K .......... .......... .......... .......... .......... 7% 23.7M 6s 2024-09-16T12:22:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 12350K .......... .......... .......... .......... .......... 7% 24.0M 6s 2024-09-16T12:22:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 12400K .......... .......... .......... .......... .......... 7% 38.4M 6s 2024-09-16T12:22:54.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 12450K .......... .......... .......... .......... .......... 7% 27.9M 6s 2024-09-16T12:22:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 12500K .......... .......... .......... .......... .......... 7% 21.7M 6s 2024-09-16T12:22:54.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 12550K .......... .......... .......... .......... .......... 7% 44.8M 6s 2024-09-16T12:22:54.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 12600K .......... .......... .......... .......... .......... 7% 23.8M 6s 2024-09-16T12:22:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 12650K .......... .......... .......... .......... .......... 7% 36.0M 6s 2024-09-16T12:22:54.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 12700K .......... .......... .......... .......... .......... 7% 25.1M 6s 2024-09-16T12:22:54.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 12750K .......... .......... .......... .......... .......... 7% 22.8M 6s 2024-09-16T12:22:54.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 12800K .......... .......... .......... .......... .......... 7% 47.7M 6s 2024-09-16T12:22:54.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 12850K .......... .......... .......... .......... .......... 7% 27.7M 6s 2024-09-16T12:22:54.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 12900K .......... .......... .......... .......... .......... 7% 32.4M 6s 2024-09-16T12:22:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 12950K .......... .......... .......... .......... .......... 7% 23.1M 6s 2024-09-16T12:22:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 13000K .......... .......... .......... .......... .......... 7% 69.5M 6s 2024-09-16T12:22:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 13050K .......... .......... .......... .......... .......... 7% 28.8M 6s 2024-09-16T12:22:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 13100K .......... .......... .......... .......... .......... 7% 23.9M 6s 2024-09-16T12:22:54.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 13150K .......... .......... .......... .......... .......... 7% 24.3M 6s 2024-09-16T12:22:54.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 13200K .......... .......... .......... .......... .......... 7% 25.2M 6s 2024-09-16T12:22:54.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 13250K .......... .......... .......... .......... .......... 7% 22.7M 6s 2024-09-16T12:22:54.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 13300K .......... .......... .......... .......... .......... 7% 50.4M 6s 2024-09-16T12:22:54.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 13350K .......... .......... .......... .......... .......... 7% 20.3M 6s 2024-09-16T12:22:54.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 13400K .......... .......... .......... .......... .......... 7% 36.8M 6s 2024-09-16T12:22:54.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 13450K .......... .......... .......... .......... .......... 8% 26.6M 6s 2024-09-16T12:22:54.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 13500K .......... .......... .......... .......... .......... 8% 44.4M 6s 2024-09-16T12:22:54.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 13550K .......... .......... .......... .......... .......... 8% 22.3M 6s 2024-09-16T12:22:54.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 13600K .......... .......... .......... .......... .......... 8% 27.6M 6s 2024-09-16T12:22:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 13650K .......... .......... .......... .......... .......... 8% 23.2M 6s 2024-09-16T12:22:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 13700K .......... .......... .......... .......... .......... 8% 34.6M 6s 2024-09-16T12:22:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 13750K .......... .......... .......... .......... .......... 8% 33.7M 6s 2024-09-16T12:22:54.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 13800K .......... .......... .......... .......... .......... 8% 32.9M 6s 2024-09-16T12:22:54.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 13850K .......... .......... .......... .......... .......... 8% 24.3M 6s 2024-09-16T12:22:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 13900K .......... .......... .......... .......... .......... 8% 33.6M 6s 2024-09-16T12:22:54.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 13950K .......... .......... .......... .......... .......... 8% 24.1M 6s 2024-09-16T12:22:54.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 14000K .......... .......... .......... .......... .......... 8% 28.3M 6s 2024-09-16T12:22:54.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 14050K .......... .......... .......... .......... .......... 8% 41.3M 6s 2024-09-16T12:22:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 14100K .......... .......... .......... .......... .......... 8% 25.4M 6s 2024-09-16T12:22:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 14150K .......... .......... .......... .......... .......... 8% 35.6M 6s 2024-09-16T12:22:54.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 14200K .......... .......... .......... .......... .......... 8% 22.1M 6s 2024-09-16T12:22:54.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 14250K .......... .......... .......... .......... .......... 8% 24.9M 6s 2024-09-16T12:22:54.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 14300K .......... .......... .......... .......... .......... 8% 49.7M 6s 2024-09-16T12:22:54.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 14350K .......... .......... .......... .......... .......... 8% 25.2M 6s 2024-09-16T12:22:54.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 14400K .......... .......... .......... .......... .......... 8% 23.0M 6s 2024-09-16T12:22:54.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 14450K .......... .......... .......... .......... .......... 8% 36.7M 6s 2024-09-16T12:22:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 14500K .......... .......... .......... .......... .......... 8% 24.9M 6s 2024-09-16T12:22:54.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 14550K .......... .......... .......... .......... .......... 8% 47.4M 6s 2024-09-16T12:22:54.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 14600K .......... .......... .......... .......... .......... 8% 25.3M 6s 2024-09-16T12:22:54.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 14650K .......... .......... .......... .......... .......... 8% 37.2M 6s 2024-09-16T12:22:54.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 14700K .......... .......... .......... .......... .......... 8% 23.3M 6s 2024-09-16T12:22:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 14750K .......... .......... .......... .......... .......... 8% 23.5M 6s 2024-09-16T12:22:54.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 14800K .......... .......... .......... .......... .......... 8% 24.5M 6s 2024-09-16T12:22:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 14850K .......... .......... .......... .......... .......... 8% 54.4M 6s 2024-09-16T12:22:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 14900K .......... .......... .......... .......... .......... 8% 32.2M 6s 2024-09-16T12:22:54.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 14950K .......... .......... .......... .......... .......... 8% 29.0M 6s 2024-09-16T12:22:54.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 15000K .......... .......... .......... .......... .......... 8% 21.3M 6s 2024-09-16T12:22:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 15050K .......... .......... .......... .......... .......... 8% 50.9M 6s 2024-09-16T12:22:54.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 15100K .......... .......... .......... .......... .......... 8% 24.1M 6s 2024-09-16T12:22:54.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 15150K .......... .......... .......... .......... .......... 9% 23.8M 6s 2024-09-16T12:22:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 15200K .......... .......... .......... .......... .......... 9% 23.7M 6s 2024-09-16T12:22:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 15250K .......... .......... .......... .......... .......... 9% 39.0M 6s 2024-09-16T12:22:54.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 15300K .......... .......... .......... .......... .......... 9% 48.0M 6s 2024-09-16T12:22:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 15350K .......... .......... .......... .......... .......... 9% 24.6M 6s 2024-09-16T12:22:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 15400K .......... .......... .......... .......... .......... 9% 32.3M 6s 2024-09-16T12:22:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 15450K .......... .......... .......... .......... .......... 9% 24.9M 6s 2024-09-16T12:22:54.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 15500K .......... .......... .......... .......... .......... 9% 24.7M 6s 2024-09-16T12:22:54.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 15550K .......... .......... .......... .......... .......... 9% 24.0M 6s 2024-09-16T12:22:54.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 15600K .......... .......... .......... .......... .......... 9% 51.1M 6s 2024-09-16T12:22:54.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 15650K .......... .......... .......... .......... .......... 9% 24.8M 6s 2024-09-16T12:22:54.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 15700K .......... .......... .......... .......... .......... 9% 33.3M 6s 2024-09-16T12:22:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 15750K .......... .......... .......... .......... .......... 9% 24.2M 6s 2024-09-16T12:22:54.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 15800K .......... .......... .......... .......... .......... 9% 48.1M 6s 2024-09-16T12:22:54.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 15850K .......... .......... .......... .......... .......... 9% 24.8M 6s 2024-09-16T12:22:54.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 15900K .......... .......... .......... .......... .......... 9% 36.1M 6s 2024-09-16T12:22:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 15950K .......... .......... .......... .......... .......... 9% 23.6M 6s 2024-09-16T12:22:54.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 16000K .......... .......... .......... .......... .......... 9% 24.8M 6s 2024-09-16T12:22:54.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 16050K .......... .......... .......... .......... .......... 9% 24.4M 6s 2024-09-16T12:22:54.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 16100K .......... .......... .......... .......... .......... 9% 47.6M 6s 2024-09-16T12:22:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 16150K .......... .......... .......... .......... .......... 9% 24.3M 6s 2024-09-16T12:22:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 16200K .......... .......... .......... .......... .......... 9% 36.5M 6s 2024-09-16T12:22:54.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 16250K .......... .......... .......... .......... .......... 9% 24.9M 6s 2024-09-16T12:22:54.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 16300K .......... .......... .......... .......... .......... 9% 47.4M 6s 2024-09-16T12:22:54.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 16350K .......... .......... .......... .......... .......... 9% 25.3M 6s 2024-09-16T12:22:54.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 16400K .......... .......... .......... .......... .......... 9% 23.5M 6s 2024-09-16T12:22:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 16450K .......... .......... .......... .......... .......... 9% 35.6M 6s 2024-09-16T12:22:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 16500K .......... .......... .......... .......... .......... 9% 24.0M 6s 2024-09-16T12:22:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 16550K .......... .......... .......... .......... .......... 9% 24.1M 6s 2024-09-16T12:22:54.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 16600K .......... .......... .......... .......... .......... 9% 53.9M 6s 2024-09-16T12:22:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 16650K .......... .......... .......... .......... .......... 9% 24.2M 6s 2024-09-16T12:22:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 16700K .......... .......... .......... .......... .......... 9% 40.3M 6s 2024-09-16T12:22:54.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 16750K .......... .......... .......... .......... .......... 9% 22.0M 6s 2024-09-16T12:22:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 16800K .......... .......... .......... .......... .......... 9% 24.9M 6s 2024-09-16T12:22:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 16850K .......... .......... .......... .......... .......... 10% 47.3M 6s 2024-09-16T12:22:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 16900K .......... .......... .......... .......... .......... 10% 25.5M 6s 2024-09-16T12:22:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 16950K .......... .......... .......... .......... .......... 10% 34.5M 6s 2024-09-16T12:22:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 17000K .......... .......... .......... .......... .......... 10% 24.5M 6s 2024-09-16T12:22:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 17050K .......... .......... .......... .......... .......... 10% 22.0M 6s 2024-09-16T12:22:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 17100K .......... .......... .......... .......... .......... 10% 62.1M 6s 2024-09-16T12:22:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 17150K .......... .......... .......... .......... .......... 10% 22.5M 6s 2024-09-16T12:22:54.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 17200K .......... .......... .......... .......... .......... 10% 23.5M 6s 2024-09-16T12:22:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 17250K .......... .......... .......... .......... .......... 10% 32.9M 6s 2024-09-16T12:22:54.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 17300K .......... .......... .......... .......... .......... 10% 24.9M 6s 2024-09-16T12:22:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 17350K .......... .......... .......... .......... .......... 10% 45.5M 6s 2024-09-16T12:22:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 17400K .......... .......... .......... .......... .......... 10% 24.6M 6s 2024-09-16T12:22:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 17450K .......... .......... .......... .......... .......... 10% 40.0M 6s 2024-09-16T12:22:54.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 17500K .......... .......... .......... .......... .......... 10% 23.7M 6s 2024-09-16T12:22:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 17550K .......... .......... .......... .......... .......... 10% 35.5M 6s 2024-09-16T12:22:54.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 17600K .......... .......... .......... .......... .......... 10% 21.5M 6s 2024-09-16T12:22:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 17650K .......... .......... .......... .......... .......... 10% 34.2M 6s 2024-09-16T12:22:54.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 17700K .......... .......... .......... .......... .......... 10% 33.9M 6s 2024-09-16T12:22:54.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 17750K .......... .......... .......... .......... .......... 10% 26.3M 6s 2024-09-16T12:22:54.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 17800K .......... .......... .......... .......... .......... 10% 23.8M 6s 2024-09-16T12:22:54.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 17850K .......... .......... .......... .......... .......... 10% 52.2M 6s 2024-09-16T12:22:54.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 17900K .......... .......... .......... .......... .......... 10% 23.9M 6s 2024-09-16T12:22:54.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 17950K .......... .......... .......... .......... .......... 10% 23.8M 6s 2024-09-16T12:22:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 18000K .......... .......... .......... .......... .......... 10% 32.1M 6s 2024-09-16T12:22:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 18050K .......... .......... .......... .......... .......... 10% 25.1M 6s 2024-09-16T12:22:54.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 18100K .......... .......... .......... .......... .......... 10% 48.3M 6s 2024-09-16T12:22:54.360 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 18150K .......... .......... .......... .......... .......... 10% 24.3M 6s 2024-09-16T12:22:54.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 18200K .......... .......... .......... .......... .......... 10% 32.9M 6s 2024-09-16T12:22:54.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 18250K .......... .......... .......... .......... .......... 10% 25.1M 6s 2024-09-16T12:22:54.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 18300K .......... .......... .......... .......... .......... 10% 23.7M 6s 2024-09-16T12:22:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 18350K .......... .......... .......... .......... .......... 10% 22.8M 6s 2024-09-16T12:22:54.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 18400K .......... .......... .......... .......... .......... 10% 48.2M 6s 2024-09-16T12:22:54.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 18450K .......... .......... .......... .......... .......... 10% 25.7M 6s 2024-09-16T12:22:54.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 18500K .......... .......... .......... .......... .......... 11% 35.0M 6s 2024-09-16T12:22:54.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 18550K .......... .......... .......... .......... .......... 11% 24.1M 6s 2024-09-16T12:22:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 18600K .......... .......... .......... .......... .......... 11% 49.5M 6s 2024-09-16T12:22:54.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 18650K .......... .......... .......... .......... .......... 11% 24.4M 6s 2024-09-16T12:22:54.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 18700K .......... .......... .......... .......... .......... 11% 36.4M 6s 2024-09-16T12:22:54.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 18750K .......... .......... .......... .......... .......... 11% 22.8M 6s 2024-09-16T12:22:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 18800K .......... .......... .......... .......... .......... 11% 25.5M 6s 2024-09-16T12:22:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 18850K .......... .......... .......... .......... .......... 11% 25.5M 6s 2024-09-16T12:22:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 18900K .......... .......... .......... .......... .......... 11% 47.1M 6s 2024-09-16T12:22:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 18950K .......... .......... .......... .......... .......... 11% 23.2M 6s 2024-09-16T12:22:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 19000K .......... .......... .......... .......... .......... 11% 44.9M 6s 2024-09-16T12:22:54.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 19050K .......... .......... .......... .......... .......... 11% 24.5M 6s 2024-09-16T12:22:54.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 19100K .......... .......... .......... .......... .......... 11% 40.5M 6s 2024-09-16T12:22:54.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 19150K .......... .......... .......... .......... .......... 11% 23.8M 6s 2024-09-16T12:22:54.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 19200K .......... .......... .......... .......... .......... 11% 25.3M 6s 2024-09-16T12:22:54.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 19250K .......... .......... .......... .......... .......... 11% 33.2M 6s 2024-09-16T12:22:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 19300K .......... .......... .......... .......... .......... 11% 24.0M 6s 2024-09-16T12:22:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 19350K .......... .......... .......... .......... .......... 11% 23.5M 6s 2024-09-16T12:22:54.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 19400K .......... .......... .......... .......... .......... 11% 54.4M 6s 2024-09-16T12:22:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 19450K .......... .......... .......... .......... .......... 11% 34.0M 6s 2024-09-16T12:22:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 19500K .......... .......... .......... .......... .......... 11% 26.1M 6s 2024-09-16T12:22:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 19550K .......... .......... .......... .......... .......... 11% 22.8M 6s 2024-09-16T12:22:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 19600K .......... .......... .......... .......... .......... 11% 24.3M 6s 2024-09-16T12:22:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 19650K .......... .......... .......... .......... .......... 11% 49.7M 6s 2024-09-16T12:22:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 19700K .......... .......... .......... .......... .......... 11% 33.3M 6s 2024-09-16T12:22:54.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 19750K .......... .......... .......... .......... .......... 11% 25.6M 6s 2024-09-16T12:22:54.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 19800K .......... .......... .......... .......... .......... 11% 24.9M 6s 2024-09-16T12:22:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 19850K .......... .......... .......... .......... .......... 11% 23.5M 6s 2024-09-16T12:22:54.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 19900K .......... .......... .......... .......... .......... 11% 58.9M 5s 2024-09-16T12:22:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 19950K .......... .......... .......... .......... .......... 11% 24.0M 5s 2024-09-16T12:22:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 20000K .......... .......... .......... .......... .......... 11% 22.9M 5s 2024-09-16T12:22:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 20050K .......... .......... .......... .......... .......... 11% 32.1M 5s 2024-09-16T12:22:54.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 20100K .......... .......... .......... .......... .......... 11% 27.1M 5s 2024-09-16T12:22:54.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 20150K .......... .......... .......... .......... .......... 11% 45.4M 5s 2024-09-16T12:22:54.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 20200K .......... .......... .......... .......... .......... 12% 26.4M 5s 2024-09-16T12:22:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 20250K .......... .......... .......... .......... .......... 12% 35.0M 5s 2024-09-16T12:22:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 20300K .......... .......... .......... .......... .......... 12% 24.6M 5s 2024-09-16T12:22:54.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 20350K .......... .......... .......... .......... .......... 12% 23.7M 5s 2024-09-16T12:22:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 20400K .......... .......... .......... .......... .......... 12% 23.5M 5s 2024-09-16T12:22:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 20450K .......... .......... .......... .......... .......... 12% 58.0M 5s 2024-09-16T12:22:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 20500K .......... .......... .......... .......... .......... 12% 32.7M 5s 2024-09-16T12:22:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 20550K .......... .......... .......... .......... .......... 12% 24.5M 5s 2024-09-16T12:22:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 20600K .......... .......... .......... .......... .......... 12% 23.9M 5s 2024-09-16T12:22:54.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 20650K .......... .......... .......... .......... .......... 12% 51.2M 5s 2024-09-16T12:22:54.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 20700K .......... .......... .......... .......... .......... 12% 24.2M 5s 2024-09-16T12:22:54.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 20750K .......... .......... .......... .......... .......... 12% 23.9M 5s 2024-09-16T12:22:54.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 20800K .......... .......... .......... .......... .......... 12% 26.3M 5s 2024-09-16T12:22:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 20850K .......... .......... .......... .......... .......... 12% 35.1M 5s 2024-09-16T12:22:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 20900K .......... .......... .......... .......... .......... 12% 47.5M 5s 2024-09-16T12:22:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 20950K .......... .......... .......... .......... .......... 12% 23.8M 5s 2024-09-16T12:22:54.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 21000K .......... .......... .......... .......... .......... 12% 36.5M 5s 2024-09-16T12:22:54.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 21050K .......... .......... .......... .......... .......... 12% 21.6M 5s 2024-09-16T12:22:54.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 21100K .......... .......... .......... .......... .......... 12% 25.2M 5s 2024-09-16T12:22:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 21150K .......... .......... .......... .......... .......... 12% 23.6M 5s 2024-09-16T12:22:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 21200K .......... .......... .......... .......... .......... 12% 52.7M 5s 2024-09-16T12:22:54.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 21250K .......... .......... .......... .......... .......... 12% 24.3M 5s 2024-09-16T12:22:54.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 21300K .......... .......... .......... .......... .......... 12% 26.5M 5s 2024-09-16T12:22:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 21350K .......... .......... .......... .......... .......... 12% 38.5M 5s 2024-09-16T12:22:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 21400K .......... .......... .......... .......... .......... 12% 40.0M 5s 2024-09-16T12:22:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 21450K .......... .......... .......... .......... .......... 12% 24.4M 5s 2024-09-16T12:22:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 21500K .......... .......... .......... .......... .......... 12% 37.4M 5s 2024-09-16T12:22:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 21550K .......... .......... .......... .......... .......... 12% 24.6M 5s 2024-09-16T12:22:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 21600K .......... .......... .......... .......... .......... 12% 22.4M 5s 2024-09-16T12:22:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 21650K .......... .......... .......... .......... .......... 12% 25.4M 5s 2024-09-16T12:22:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 21700K .......... .......... .......... .......... .......... 12% 59.0M 5s 2024-09-16T12:22:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 21750K .......... .......... .......... .......... .......... 12% 21.9M 5s 2024-09-16T12:22:54.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 21800K .......... .......... .......... .......... .......... 12% 35.3M 5s 2024-09-16T12:22:54.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 21850K .......... .......... .......... .......... .......... 12% 25.5M 5s 2024-09-16T12:22:54.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 21900K .......... .......... .......... .......... .......... 13% 51.1M 5s 2024-09-16T12:22:54.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 21950K .......... .......... .......... .......... .......... 13% 23.5M 5s 2024-09-16T12:22:54.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 22000K .......... .......... .......... .......... .......... 13% 25.0M 5s 2024-09-16T12:22:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 22050K .......... .......... .......... .......... .......... 13% 36.6M 5s 2024-09-16T12:22:54.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 22100K .......... .......... .......... .......... .......... 13% 25.0M 5s 2024-09-16T12:22:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 22150K .......... .......... .......... .......... .......... 13% 21.9M 5s 2024-09-16T12:22:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 22200K .......... .......... .......... .......... .......... 13% 61.3M 5s 2024-09-16T12:22:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 22250K .......... .......... .......... .......... .......... 13% 35.1M 5s 2024-09-16T12:22:54.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 22300K .......... .......... .......... .......... .......... 13% 23.5M 5s 2024-09-16T12:22:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 22350K .......... .......... .......... .......... .......... 13% 23.1M 5s 2024-09-16T12:22:54.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 22400K .......... .......... .......... .......... .......... 13% 25.4M 5s 2024-09-16T12:22:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 22450K .......... .......... .......... .......... .......... 13% 49.2M 5s 2024-09-16T12:22:54.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 22500K .......... .......... .......... .......... .......... 13% 24.8M 5s 2024-09-16T12:22:54.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 22550K .......... .......... .......... .......... .......... 13% 35.1M 5s 2024-09-16T12:22:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 22600K .......... .......... .......... .......... .......... 13% 24.1M 5s 2024-09-16T12:22:54.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 22650K .......... .......... .......... .......... .......... 13% 48.7M 5s 2024-09-16T12:22:54.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 22700K .......... .......... .......... .......... .......... 13% 25.0M 5s 2024-09-16T12:22:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 22750K .......... .......... .......... .......... .......... 13% 24.2M 5s 2024-09-16T12:22:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 22800K .......... .......... .......... .......... .......... 13% 30.9M 5s 2024-09-16T12:22:54.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 22850K .......... .......... .......... .......... .......... 13% 24.6M 5s 2024-09-16T12:22:54.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 22900K .......... .......... .......... .......... .......... 13% 26.2M 5s 2024-09-16T12:22:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 22950K .......... .......... .......... .......... .......... 13% 47.2M 5s 2024-09-16T12:22:54.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 23000K .......... .......... .......... .......... .......... 13% 24.5M 5s 2024-09-16T12:22:54.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 23050K .......... .......... .......... .......... .......... 13% 35.0M 5s 2024-09-16T12:22:54.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 23100K .......... .......... .......... .......... .......... 13% 25.5M 5s 2024-09-16T12:22:54.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 23150K .......... .......... .......... .......... .......... 13% 25.7M 5s 2024-09-16T12:22:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 23200K .......... .......... .......... .......... .......... 13% 41.8M 5s 2024-09-16T12:22:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 23250K .......... .......... .......... .......... .......... 13% 25.2M 5s 2024-09-16T12:22:54.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 23300K .......... .......... .......... .......... .......... 13% 27.2M 5s 2024-09-16T12:22:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 23350K .......... .......... .......... .......... .......... 13% 28.3M 5s 2024-09-16T12:22:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 23400K .......... .......... .......... .......... .......... 13% 25.0M 5s 2024-09-16T12:22:54.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 23450K .......... .......... .......... .......... .......... 13% 53.6M 5s 2024-09-16T12:22:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 23500K .......... .......... .......... .......... .......... 13% 24.2M 5s 2024-09-16T12:22:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 23550K .......... .......... .......... .......... .......... 14% 24.7M 5s 2024-09-16T12:22:54.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 23600K .......... .......... .......... .......... .......... 14% 35.5M 5s 2024-09-16T12:22:54.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 23650K .......... .......... .......... .......... .......... 14% 23.6M 5s 2024-09-16T12:22:54.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 23700K .......... .......... .......... .......... .......... 14% 50.6M 5s 2024-09-16T12:22:54.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 23750K .......... .......... .......... .......... .......... 14% 25.4M 5s 2024-09-16T12:22:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 23800K .......... .......... .......... .......... .......... 14% 32.8M 5s 2024-09-16T12:22:54.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 23850K .......... .......... .......... .......... .......... 14% 24.5M 5s 2024-09-16T12:22:54.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 23900K .......... .......... .......... .......... .......... 14% 24.5M 5s 2024-09-16T12:22:54.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 23950K .......... .......... .......... .......... .......... 14% 33.8M 5s 2024-09-16T12:22:54.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 24000K .......... .......... .......... .......... .......... 14% 31.2M 5s 2024-09-16T12:22:54.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 24050K .......... .......... .......... .......... .......... 14% 25.0M 5s 2024-09-16T12:22:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 24100K .......... .......... .......... .......... .......... 14% 33.3M 5s 2024-09-16T12:22:54.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 24150K .......... .......... .......... .......... .......... 14% 24.9M 5s 2024-09-16T12:22:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 24200K .......... .......... .......... .......... .......... 14% 48.2M 5s 2024-09-16T12:22:54.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 24250K .......... .......... .......... .......... .......... 14% 27.2M 5s 2024-09-16T12:22:54.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 24300K .......... .......... .......... .......... .......... 14% 20.7M 5s 2024-09-16T12:22:54.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 24350K .......... .......... .......... .......... .......... 14% 36.5M 5s 2024-09-16T12:22:54.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 24400K .......... .......... .......... .......... .......... 14% 23.9M 5s 2024-09-16T12:22:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 24450K .......... .......... .......... .......... .......... 14% 25.8M 5s 2024-09-16T12:22:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 24500K .......... .......... .......... .......... .......... 14% 54.5M 5s 2024-09-16T12:22:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 24550K .......... .......... .......... .......... .......... 14% 31.5M 5s 2024-09-16T12:22:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 24600K .......... .......... .......... .......... .......... 14% 25.1M 5s 2024-09-16T12:22:54.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 24650K .......... .......... .......... .......... .......... 14% 24.4M 5s 2024-09-16T12:22:54.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 24700K .......... .......... .......... .......... .......... 14% 49.6M 5s 2024-09-16T12:22:54.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 24750K .......... .......... .......... .......... .......... 14% 24.2M 5s 2024-09-16T12:22:54.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 24800K .......... .......... .......... .......... .......... 14% 22.2M 5s 2024-09-16T12:22:54.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 24850K .......... .......... .......... .......... .......... 14% 30.1M 5s 2024-09-16T12:22:54.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 24900K .......... .......... .......... .......... .......... 14% 25.8M 5s 2024-09-16T12:22:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 24950K .......... .......... .......... .......... .......... 14% 18.3M 5s 2024-09-16T12:22:54.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 25000K .......... .......... .......... .......... .......... 14% 58.5M 5s 2024-09-16T12:22:54.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 25050K .......... .......... .......... .......... .......... 14% 32.2M 5s 2024-09-16T12:22:54.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 25100K .......... .......... .......... .......... .......... 14% 24.2M 5s 2024-09-16T12:22:54.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 25150K .......... .......... .......... .......... .......... 14% 24.2M 5s 2024-09-16T12:22:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 25200K .......... .......... .......... .......... .......... 14% 25.5M 5s 2024-09-16T12:22:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 25250K .......... .......... .......... .......... .......... 15% 49.0M 5s 2024-09-16T12:22:54.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 25300K .......... .......... .......... .......... .......... 15% 25.1M 5s 2024-09-16T12:22:54.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 25350K .......... .......... .......... .......... .......... 15% 34.2M 5s 2024-09-16T12:22:54.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 25400K .......... .......... .......... .......... .......... 15% 25.4M 5s 2024-09-16T12:22:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 25450K .......... .......... .......... .......... .......... 15% 45.3M 5s 2024-09-16T12:22:54.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 25500K .......... .......... .......... .......... .......... 15% 25.0M 5s 2024-09-16T12:22:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 25550K .......... .......... .......... .......... .......... 15% 24.0M 5s 2024-09-16T12:22:54.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 25600K .......... .......... .......... .......... .......... 15% 24.0M 5s 2024-09-16T12:22:54.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 25650K .......... .......... .......... .......... .......... 15% 34.3M 5s 2024-09-16T12:22:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 25700K .......... .......... .......... .......... .......... 15% 30.3M 5s 2024-09-16T12:22:54.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 25750K .......... .......... .......... .......... .......... 15% 37.1M 5s 2024-09-16T12:22:54.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 25800K .......... .......... .......... .......... .......... 15% 28.2M 5s 2024-09-16T12:22:54.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 25850K .......... .......... .......... .......... .......... 15% 30.2M 5s 2024-09-16T12:22:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 25900K .......... .......... .......... .......... .......... 15% 24.6M 5s 2024-09-16T12:22:54.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 25950K .......... .......... .......... .......... .......... 15% 23.1M 5s 2024-09-16T12:22:54.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 26000K .......... .......... .......... .......... .......... 15% 48.8M 5s 2024-09-16T12:22:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 26050K .......... .......... .......... .......... .......... 15% 23.0M 5s 2024-09-16T12:22:54.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 26100K .......... .......... .......... .......... .......... 15% 36.3M 5s 2024-09-16T12:22:54.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 26150K .......... .......... .......... .......... .......... 15% 23.9M 5s 2024-09-16T12:22:54.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 26200K .......... .......... .......... .......... .......... 15% 25.1M 5s 2024-09-16T12:22:54.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 26250K .......... .......... .......... .......... .......... 15% 54.6M 5s 2024-09-16T12:22:54.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 26300K .......... .......... .......... .......... .......... 15% 23.6M 5s 2024-09-16T12:22:54.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 26350K .......... .......... .......... .......... .......... 15% 25.0M 5s 2024-09-16T12:22:54.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 26400K .......... .......... .......... .......... .......... 15% 37.2M 5s 2024-09-16T12:22:54.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 26450K .......... .......... .......... .......... .......... 15% 23.6M 5s 2024-09-16T12:22:54.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 26500K .......... .......... .......... .......... .......... 15% 54.3M 5s 2024-09-16T12:22:54.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 26550K .......... .......... .......... .......... .......... 15% 23.1M 5s 2024-09-16T12:22:54.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 26600K .......... .......... .......... .......... .......... 15% 36.5M 5s 2024-09-16T12:22:54.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 26650K .......... .......... .......... .......... .......... 15% 24.0M 5s 2024-09-16T12:22:54.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 26700K .......... .......... .......... .......... .......... 15% 23.6M 5s 2024-09-16T12:22:54.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 26750K .......... .......... .......... .......... .......... 15% 29.4M 5s 2024-09-16T12:22:54.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 26800K .......... .......... .......... .......... .......... 15% 35.8M 5s 2024-09-16T12:22:54.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 26850K .......... .......... .......... .......... .......... 15% 25.4M 5s 2024-09-16T12:22:54.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 26900K .......... .......... .......... .......... .......... 15% 34.0M 5s 2024-09-16T12:22:54.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 26950K .......... .......... .......... .......... .......... 16% 23.3M 5s 2024-09-16T12:22:54.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 27000K .......... .......... .......... .......... .......... 16% 53.7M 5s 2024-09-16T12:22:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 27050K .......... .......... .......... .......... .......... 16% 25.4M 5s 2024-09-16T12:22:54.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 27100K .......... .......... .......... .......... .......... 16% 34.4M 5s 2024-09-16T12:22:54.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 27150K .......... .......... .......... .......... .......... 16% 24.7M 5s 2024-09-16T12:22:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 27200K .......... .......... .......... .......... .......... 16% 23.2M 5s 2024-09-16T12:22:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 27250K .......... .......... .......... .......... .......... 16% 24.7M 5s 2024-09-16T12:22:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 27300K .......... .......... .......... .......... .......... 16% 57.5M 5s 2024-09-16T12:22:54.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 27350K .......... .......... .......... .......... .......... 16% 22.7M 5s 2024-09-16T12:22:54.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 27400K .......... .......... .......... .......... .......... 16% 34.7M 5s 2024-09-16T12:22:54.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 27450K .......... .......... .......... .......... .......... 16% 25.0M 5s 2024-09-16T12:22:54.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 27500K .......... .......... .......... .......... .......... 16% 49.4M 5s 2024-09-16T12:22:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 27550K .......... .......... .......... .......... .......... 16% 23.7M 5s 2024-09-16T12:22:54.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 27600K .......... .......... .......... .......... .......... 16% 25.4M 5s 2024-09-16T12:22:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 27650K .......... .......... .......... .......... .......... 16% 37.4M 5s 2024-09-16T12:22:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 27700K .......... .......... .......... .......... .......... 16% 25.4M 5s 2024-09-16T12:22:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 27750K .......... .......... .......... .......... .......... 16% 45.5M 5s 2024-09-16T12:22:54.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 27800K .......... .......... .......... .......... .......... 16% 24.5M 5s 2024-09-16T12:22:54.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 27850K .......... .......... .......... .......... .......... 16% 33.4M 5s 2024-09-16T12:22:54.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 27900K .......... .......... .......... .......... .......... 16% 25.1M 5s 2024-09-16T12:22:54.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 27950K .......... .......... .......... .......... .......... 16% 23.7M 5s 2024-09-16T12:22:54.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 28000K .......... .......... .......... .......... .......... 16% 26.8M 5s 2024-09-16T12:22:54.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 28050K .......... .......... .......... .......... .......... 16% 42.7M 5s 2024-09-16T12:22:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 28100K .......... .......... .......... .......... .......... 16% 24.8M 5s 2024-09-16T12:22:54.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 28150K .......... .......... .......... .......... .......... 16% 35.6M 5s 2024-09-16T12:22:54.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 28200K .......... .......... .......... .......... .......... 16% 24.6M 5s 2024-09-16T12:22:54.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 28250K .......... .......... .......... .......... .......... 16% 50.2M 5s 2024-09-16T12:22:54.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 28300K .......... .......... .......... .......... .......... 16% 24.8M 5s 2024-09-16T12:22:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 28350K .......... .......... .......... .......... .......... 16% 24.8M 5s 2024-09-16T12:22:54.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 28400K .......... .......... .......... .......... .......... 16% 32.3M 5s 2024-09-16T12:22:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 28450K .......... .......... .......... .......... .......... 16% 24.0M 5s 2024-09-16T12:22:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 28500K .......... .......... .......... .......... .......... 16% 24.1M 5s 2024-09-16T12:22:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 28550K .......... .......... .......... .......... .......... 16% 61.8M 5s 2024-09-16T12:22:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 28600K .......... .......... .......... .......... .......... 16% 22.0M 5s 2024-09-16T12:22:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 28650K .......... .......... .......... .......... .......... 17% 36.4M 5s 2024-09-16T12:22:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 28700K .......... .......... .......... .......... .......... 17% 24.5M 5s 2024-09-16T12:22:54.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 28750K .......... .......... .......... .......... .......... 17% 23.7M 5s 2024-09-16T12:22:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 28800K .......... .......... .......... .......... .......... 17% 34.1M 5s 2024-09-16T12:22:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 28850K .......... .......... .......... .......... .......... 17% 24.5M 5s 2024-09-16T12:22:54.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 28900K .......... .......... .......... .......... .......... 17% 35.2M 5s 2024-09-16T12:22:54.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 28950K .......... .......... .......... .......... .......... 17% 23.9M 5s 2024-09-16T12:22:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 29000K .......... .......... .......... .......... .......... 17% 24.4M 5s 2024-09-16T12:22:54.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 29050K .......... .......... .......... .......... .......... 17% 52.5M 5s 2024-09-16T12:22:54.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 29100K .......... .......... .......... .......... .......... 17% 24.1M 5s 2024-09-16T12:22:54.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 29150K .......... .......... .......... .......... .......... 17% 20.3M 5s 2024-09-16T12:22:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 29200K .......... .......... .......... .......... .......... 17% 29.6M 5s 2024-09-16T12:22:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 29250K .......... .......... .......... .......... .......... 17% 26.5M 5s 2024-09-16T12:22:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 29300K .......... .......... .......... .......... .......... 17% 47.4M 5s 2024-09-16T12:22:54.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 29350K .......... .......... .......... .......... .......... 17% 24.4M 5s 2024-09-16T12:22:54.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 29400K .......... .......... .......... .......... .......... 17% 32.6M 5s 2024-09-16T12:22:54.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 29450K .......... .......... .......... .......... .......... 17% 25.2M 5s 2024-09-16T12:22:54.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 29500K .......... .......... .......... .......... .......... 17% 22.9M 5s 2024-09-16T12:22:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 29550K .......... .......... .......... .......... .......... 17% 24.7M 5s 2024-09-16T12:22:54.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 29600K .......... .......... .......... .......... .......... 17% 60.7M 5s 2024-09-16T12:22:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 29650K .......... .......... .......... .......... .......... 17% 29.3M 5s 2024-09-16T12:22:54.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 29700K .......... .......... .......... .......... .......... 17% 25.5M 5s 2024-09-16T12:22:54.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 29750K .......... .......... .......... .......... .......... 17% 25.1M 5s 2024-09-16T12:22:54.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 29800K .......... .......... .......... .......... .......... 17% 49.5M 5s 2024-09-16T12:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 29850K .......... .......... .......... .......... .......... 17% 25.1M 5s 2024-09-16T12:22:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 29900K .......... .......... .......... .......... .......... 17% 36.7M 5s 2024-09-16T12:22:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 29950K .......... .......... .......... .......... .......... 17% 22.0M 5s 2024-09-16T12:22:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 30000K .......... .......... .......... .......... .......... 17% 25.2M 5s 2024-09-16T12:22:54.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 30050K .......... .......... .......... .......... .......... 17% 23.7M 5s 2024-09-16T12:22:54.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 30100K .......... .......... .......... .......... .......... 17% 56.9M 5s 2024-09-16T12:22:54.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 30150K .......... .......... .......... .......... .......... 17% 29.2M 5s 2024-09-16T12:22:54.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 30200K .......... .......... .......... .......... .......... 17% 25.2M 5s 2024-09-16T12:22:54.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 30250K .......... .......... .......... .......... .......... 17% 25.6M 5s 2024-09-16T12:22:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 30300K .......... .......... .......... .......... .......... 18% 49.6M 5s 2024-09-16T12:22:54.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 30350K .......... .......... .......... .......... .......... 18% 23.9M 5s 2024-09-16T12:22:54.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 30400K .......... .......... .......... .......... .......... 18% 25.5M 5s 2024-09-16T12:22:54.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 30450K .......... .......... .......... .......... .......... 18% 33.7M 5s 2024-09-16T12:22:54.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 30500K .......... .......... .......... .......... .......... 18% 34.5M 5s 2024-09-16T12:22:54.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 30550K .......... .......... .......... .......... .......... 18% 29.0M 5s 2024-09-16T12:22:54.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 30600K .......... .......... .......... .......... .......... 18% 25.6M 5s 2024-09-16T12:22:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 30650K .......... .......... .......... .......... .......... 18% 22.1M 5s 2024-09-16T12:22:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 30700K .......... .......... .......... .......... .......... 18% 43.1M 5s 2024-09-16T12:22:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 30750K .......... .......... .......... .......... .......... 18% 20.7M 5s 2024-09-16T12:22:54.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 30800K .......... .......... .......... .......... .......... 18% 25.0M 5s 2024-09-16T12:22:54.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 30850K .......... .......... .......... .......... .......... 18% 50.8M 5s 2024-09-16T12:22:54.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 30900K .......... .......... .......... .......... .......... 18% 25.6M 5s 2024-09-16T12:22:54.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 30950K .......... .......... .......... .......... .......... 18% 34.1M 5s 2024-09-16T12:22:54.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 31000K .......... .......... .......... .......... .......... 18% 23.3M 5s 2024-09-16T12:22:54.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 31050K .......... .......... .......... .......... .......... 18% 51.0M 5s 2024-09-16T12:22:54.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 31100K .......... .......... .......... .......... .......... 18% 25.3M 5s 2024-09-16T12:22:54.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 31150K .......... .......... .......... .......... .......... 18% 24.3M 5s 2024-09-16T12:22:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 31200K .......... .......... .......... .......... .......... 18% 29.7M 5s 2024-09-16T12:22:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 31250K .......... .......... .......... .......... .......... 18% 25.1M 5s 2024-09-16T12:22:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 31300K .......... .......... .......... .......... .......... 18% 25.7M 5s 2024-09-16T12:22:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 31350K .......... .......... .......... .......... .......... 18% 62.5M 5s 2024-09-16T12:22:54.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 31400K .......... .......... .......... .......... .......... 18% 22.7M 5s 2024-09-16T12:22:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 31450K .......... .......... .......... .......... .......... 18% 33.5M 5s 2024-09-16T12:22:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 31500K .......... .......... .......... .......... .......... 18% 25.0M 5s 2024-09-16T12:22:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 31550K .......... .......... .......... .......... .......... 18% 25.4M 5s 2024-09-16T12:22:54.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 31600K .......... .......... .......... .......... .......... 18% 44.4M 5s 2024-09-16T12:22:54.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 31650K .......... .......... .......... .......... .......... 18% 25.7M 5s 2024-09-16T12:22:54.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 31700K .......... .......... .......... .......... .......... 18% 34.7M 5s 2024-09-16T12:22:54.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 31750K .......... .......... .......... .......... .......... 18% 25.0M 5s 2024-09-16T12:22:54.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 31800K .......... .......... .......... .......... .......... 18% 23.0M 5s 2024-09-16T12:22:54.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 31850K .......... .......... .......... .......... .......... 18% 55.1M 5s 2024-09-16T12:22:54.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 31900K .......... .......... .......... .......... .......... 18% 23.2M 5s 2024-09-16T12:22:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 31950K .......... .......... .......... .......... .......... 18% 25.2M 5s 2024-09-16T12:22:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 32000K .......... .......... .......... .......... .......... 19% 34.4M 5s 2024-09-16T12:22:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 32050K .......... .......... .......... .......... .......... 19% 26.9M 5s 2024-09-16T12:22:54.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 32100K .......... .......... .......... .......... .......... 19% 44.9M 5s 2024-09-16T12:22:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 32150K .......... .......... .......... .......... .......... 19% 24.8M 5s 2024-09-16T12:22:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 32200K .......... .......... .......... .......... .......... 19% 38.0M 5s 2024-09-16T12:22:54.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 32250K .......... .......... .......... .......... .......... 19% 24.2M 5s 2024-09-16T12:22:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 32300K .......... .......... .......... .......... .......... 19% 22.8M 5s 2024-09-16T12:22:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 32350K .......... .......... .......... .......... .......... 19% 25.6M 5s 2024-09-16T12:22:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 32400K .......... .......... .......... .......... .......... 19% 51.8M 5s 2024-09-16T12:22:54.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 32450K .......... .......... .......... .......... .......... 19% 22.0M 5s 2024-09-16T12:22:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 32500K .......... .......... .......... .......... .......... 19% 36.6M 5s 2024-09-16T12:22:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 32550K .......... .......... .......... .......... .......... 19% 23.9M 5s 2024-09-16T12:22:54.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 32600K .......... .......... .......... .......... .......... 19% 50.3M 5s 2024-09-16T12:22:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 32650K .......... .......... .......... .......... .......... 19% 23.9M 5s 2024-09-16T12:22:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 32700K .......... .......... .......... .......... .......... 19% 37.7M 5s 2024-09-16T12:22:54.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 32750K .......... .......... .......... .......... .......... 19% 23.7M 5s 2024-09-16T12:22:54.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 32800K .......... .......... .......... .......... .......... 19% 25.6M 5s 2024-09-16T12:22:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 32850K .......... .......... .......... .......... .......... 19% 47.4M 5s 2024-09-16T12:22:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 32900K .......... .......... .......... .......... .......... 19% 24.7M 5s 2024-09-16T12:22:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 32950K .......... .......... .......... .......... .......... 19% 30.5M 5s 2024-09-16T12:22:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 33000K .......... .......... .......... .......... .......... 19% 20.6M 5s 2024-09-16T12:22:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 33050K .......... .......... .......... .......... .......... 19% 596K 5s 2024-09-16T12:22:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 33100K .......... .......... .......... .......... .......... 19% 120M 5s 2024-09-16T12:22:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 33150K .......... .......... .......... .......... .......... 19% 112M 5s 2024-09-16T12:22:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 33200K .......... .......... .......... .......... .......... 19% 124M 5s 2024-09-16T12:22:54.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 33250K .......... .......... .......... .......... .......... 19% 130M 5s 2024-09-16T12:22:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 33300K .......... .......... .......... .......... .......... 19% 132M 5s 2024-09-16T12:22:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 33350K .......... .......... .......... .......... .......... 19% 113M 5s 2024-09-16T12:22:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 33400K .......... .......... .......... .......... .......... 19% 132M 5s 2024-09-16T12:22:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 33450K .......... .......... .......... .......... .......... 19% 133M 5s 2024-09-16T12:22:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 33500K .......... .......... .......... .......... .......... 19% 133M 5s 2024-09-16T12:22:54.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 33550K .......... .......... .......... .......... .......... 19% 113M 5s 2024-09-16T12:22:55.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 33600K .......... .......... .......... .......... .......... 19% 4.50M 5s 2024-09-16T12:22:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 33650K .......... .......... .......... .......... .......... 19% 138M 5s 2024-09-16T12:22:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 33700K .......... .......... .......... .......... .......... 20% 180M 5s 2024-09-16T12:22:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 33750K .......... .......... .......... .......... .......... 20% 166M 5s 2024-09-16T12:22:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 33800K .......... .......... .......... .......... .......... 20% 203M 5s 2024-09-16T12:22:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 33850K .......... .......... .......... .......... .......... 20% 196M 5s 2024-09-16T12:22:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 33900K .......... .......... .......... .......... .......... 20% 210M 5s 2024-09-16T12:22:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 33950K .......... .......... .......... .......... .......... 20% 182M 5s 2024-09-16T12:22:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 34000K .......... .......... .......... .......... .......... 20% 216M 5s 2024-09-16T12:22:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 34050K .......... .......... .......... .......... .......... 20% 177M 5s 2024-09-16T12:22:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 34100K .......... .......... .......... .......... .......... 20% 191M 5s 2024-09-16T12:22:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 34150K .......... .......... .......... .......... .......... 20% 186M 5s 2024-09-16T12:22:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 34200K .......... .......... .......... .......... .......... 20% 203M 5s 2024-09-16T12:22:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 34250K .......... .......... .......... .......... .......... 20% 187M 5s 2024-09-16T12:22:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 34300K .......... .......... .......... .......... .......... 20% 162M 5s 2024-09-16T12:22:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 34350K .......... .......... .......... .......... .......... 20% 171M 5s 2024-09-16T12:22:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 34400K .......... .......... .......... .......... .......... 20% 206M 5s 2024-09-16T12:22:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 34450K .......... .......... .......... .......... .......... 20% 201M 5s 2024-09-16T12:22:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 34500K .......... .......... .......... .......... .......... 20% 171M 5s 2024-09-16T12:22:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 34550K .......... .......... .......... .......... .......... 20% 156M 5s 2024-09-16T12:22:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 34600K .......... .......... .......... .......... .......... 20% 204M 5s 2024-09-16T12:22:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 34650K .......... .......... .......... .......... .......... 20% 201M 5s 2024-09-16T12:22:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 34700K .......... .......... .......... .......... .......... 20% 207M 5s 2024-09-16T12:22:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 34750K .......... .......... .......... .......... .......... 20% 178M 5s 2024-09-16T12:22:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 34800K .......... .......... .......... .......... .......... 20% 185M 5s 2024-09-16T12:22:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 34850K .......... .......... .......... .......... .......... 20% 191M 5s 2024-09-16T12:22:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 34900K .......... .......... .......... .......... .......... 20% 202M 5s 2024-09-16T12:22:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 34950K .......... .......... .......... .......... .......... 20% 192M 5s 2024-09-16T12:22:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 35000K .......... .......... .......... .......... .......... 20% 186M 5s 2024-09-16T12:22:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 35050K .......... .......... .......... .......... .......... 20% 202M 5s 2024-09-16T12:22:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 35100K .......... .......... .......... .......... .......... 20% 254M 5s 2024-09-16T12:22:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 35150K .......... .......... .......... .......... .......... 20% 235M 5s 2024-09-16T12:22:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 35200K .......... .......... .......... .......... .......... 20% 298M 5s 2024-09-16T12:22:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 35250K .......... .......... .......... .......... .......... 20% 300M 5s 2024-09-16T12:22:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 35300K .......... .......... .......... .......... .......... 20% 292M 5s 2024-09-16T12:22:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 35350K .......... .......... .......... .......... .......... 21% 268M 5s 2024-09-16T12:22:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 35400K .......... .......... .......... .......... .......... 21% 293M 5s 2024-09-16T12:22:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 35450K .......... .......... .......... .......... .......... 21% 269M 5s 2024-09-16T12:22:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 35500K .......... .......... .......... .......... .......... 21% 296M 5s 2024-09-16T12:22:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 35550K .......... .......... .......... .......... .......... 21% 242M 5s 2024-09-16T12:22:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 35600K .......... .......... .......... .......... .......... 21% 251M 5s 2024-09-16T12:22:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 35650K .......... .......... .......... .......... .......... 21% 303M 5s 2024-09-16T12:22:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 35700K .......... .......... .......... .......... .......... 21% 251M 5s 2024-09-16T12:22:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 35750K .......... .......... .......... .......... .......... 21% 230M 5s 2024-09-16T12:22:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 35800K .......... .......... .......... .......... .......... 21% 262M 5s 2024-09-16T12:22:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 35850K .......... .......... .......... .......... .......... 21% 246M 5s 2024-09-16T12:22:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 35900K .......... .......... .......... .......... .......... 21% 245M 5s 2024-09-16T12:22:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 35950K .......... .......... .......... .......... .......... 21% 224M 5s 2024-09-16T12:22:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 36000K .......... .......... .......... .......... .......... 21% 279M 5s 2024-09-16T12:22:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 36050K .......... .......... .......... .......... .......... 21% 259M 5s 2024-09-16T12:22:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 36100K .......... .......... .......... .......... .......... 21% 3.90M 5s 2024-09-16T12:22:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 36150K .......... .......... .......... .......... .......... 21% 92.6M 5s 2024-09-16T12:22:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 36200K .......... .......... .......... .......... .......... 21% 193M 5s 2024-09-16T12:22:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 36250K .......... .......... .......... .......... .......... 21% 194M 5s 2024-09-16T12:22:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 36300K .......... .......... .......... .......... .......... 21% 183M 5s 2024-09-16T12:22:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 36350K .......... .......... .......... .......... .......... 21% 166M 5s 2024-09-16T12:22:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 36400K .......... .......... .......... .......... .......... 21% 203M 5s 2024-09-16T12:22:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 36450K .......... .......... .......... .......... .......... 21% 63.5M 5s 2024-09-16T12:22:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 36500K .......... .......... .......... .......... .......... 21% 22.7M 5s 2024-09-16T12:22:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 36550K .......... .......... .......... .......... .......... 21% 41.0M 5s 2024-09-16T12:22:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 36600K .......... .......... .......... .......... .......... 21% 23.3M 5s 2024-09-16T12:22:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 36650K .......... .......... .......... .......... .......... 21% 49.1M 5s 2024-09-16T12:22:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 36700K .......... .......... .......... .......... .......... 21% 26.1M 5s 2024-09-16T12:22:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 36750K .......... .......... .......... .......... .......... 21% 24.3M 5s 2024-09-16T12:22:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 36800K .......... .......... .......... .......... .......... 21% 24.7M 5s 2024-09-16T12:22:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 36850K .......... .......... .......... .......... .......... 21% 36.7M 5s 2024-09-16T12:22:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 36900K .......... .......... .......... .......... .......... 21% 24.7M 5s 2024-09-16T12:22:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 36950K .......... .......... .......... .......... .......... 21% 49.8M 5s 2024-09-16T12:22:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 37000K .......... .......... .......... .......... .......... 21% 24.2M 5s 2024-09-16T12:22:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 37050K .......... .......... .......... .......... .......... 22% 38.0M 5s 2024-09-16T12:22:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 37100K .......... .......... .......... .......... .......... 22% 24.0M 5s 2024-09-16T12:22:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 37150K .......... .......... .......... .......... .......... 22% 25.0M 5s 2024-09-16T12:22:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 37200K .......... .......... .......... .......... .......... 22% 41.8M 5s 2024-09-16T12:22:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 37250K .......... .......... .......... .......... .......... 22% 25.6M 5s 2024-09-16T12:22:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 37300K .......... .......... .......... .......... .......... 22% 24.5M 5s 2024-09-16T12:22:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 37350K .......... .......... .......... .......... .......... 22% 36.8M 5s 2024-09-16T12:22:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 37400K .......... .......... .......... .......... .......... 22% 23.7M 5s 2024-09-16T12:22:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 37450K .......... .......... .......... .......... .......... 22% 50.6M 5s 2024-09-16T12:22:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 37500K .......... .......... .......... .......... .......... 22% 20.8M 5s 2024-09-16T12:22:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 37550K .......... .......... .......... .......... .......... 22% 24.2M 5s 2024-09-16T12:22:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 37600K .......... .......... .......... .......... .......... 22% 47.0M 5s 2024-09-16T12:22:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 37650K .......... .......... .......... .......... .......... 22% 20.0M 5s 2024-09-16T12:22:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 37700K .......... .......... .......... .......... .......... 22% 72.4M 5s 2024-09-16T12:22:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 37750K .......... .......... .......... .......... .......... 22% 26.2M 5s 2024-09-16T12:22:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 37800K .......... .......... .......... .......... .......... 22% 24.6M 5s 2024-09-16T12:22:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 37850K .......... .......... .......... .......... .......... 22% 19.1M 5s 2024-09-16T12:22:55.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 37900K .......... .......... .......... .......... .......... 22% 156M 5s 2024-09-16T12:22:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 37950K .......... .......... .......... .......... .......... 22% 15.8M 5s 2024-09-16T12:22:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 38000K .......... .......... .......... .......... .......... 22% 85.0M 5s 2024-09-16T12:22:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 38050K .......... .......... .......... .......... .......... 22% 21.8M 5s 2024-09-16T12:22:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 38100K .......... .......... .......... .......... .......... 22% 47.7M 5s 2024-09-16T12:22:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 38150K .......... .......... .......... .......... .......... 22% 23.8M 5s 2024-09-16T12:22:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 38200K .......... .......... .......... .......... .......... 22% 38.8M 5s 2024-09-16T12:22:55.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 38250K .......... .......... .......... .......... .......... 22% 24.3M 5s 2024-09-16T12:22:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 38300K .......... .......... .......... .......... .......... 22% 28.8M 5s 2024-09-16T12:22:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 38350K .......... .......... .......... .......... .......... 22% 33.5M 5s 2024-09-16T12:22:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 38400K .......... .......... .......... .......... .......... 22% 22.3M 5s 2024-09-16T12:22:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 38450K .......... .......... .......... .......... .......... 22% 46.6M 5s 2024-09-16T12:22:55.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 38500K .......... .......... .......... .......... .......... 22% 25.0M 5s 2024-09-16T12:22:55.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 38550K .......... .......... .......... .......... .......... 22% 33.6M 5s 2024-09-16T12:22:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 38600K .......... .......... .......... .......... .......... 22% 25.3M 5s 2024-09-16T12:22:55.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 38650K .......... .......... .......... .......... .......... 22% 25.1M 5s 2024-09-16T12:22:55.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 38700K .......... .......... .......... .......... .......... 22% 40.0M 5s 2024-09-16T12:22:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 38750K .......... .......... .......... .......... .......... 23% 26.3M 5s 2024-09-16T12:22:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 38800K .......... .......... .......... .......... .......... 23% 24.4M 5s 2024-09-16T12:22:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 38850K .......... .......... .......... .......... .......... 23% 34.0M 5s 2024-09-16T12:22:55.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 38900K .......... .......... .......... .......... .......... 23% 44.1M 5s 2024-09-16T12:22:55.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 38950K .......... .......... .......... .......... .......... 23% 29.1M 5s 2024-09-16T12:22:55.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 39000K .......... .......... .......... .......... .......... 23% 22.6M 5s 2024-09-16T12:22:55.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 39050K .......... .......... .......... .......... .......... 23% 30.4M 5s 2024-09-16T12:22:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 39100K .......... .......... .......... .......... .......... 23% 31.2M 5s 2024-09-16T12:22:55.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 39150K .......... .......... .......... .......... .......... 23% 26.6M 5s 2024-09-16T12:22:55.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 39200K .......... .......... .......... .......... .......... 23% 19.5M 5s 2024-09-16T12:22:55.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 39250K .......... .......... .......... .......... .......... 23% 50.1M 5s 2024-09-16T12:22:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 39300K .......... .......... .......... .......... .......... 23% 27.0M 5s 2024-09-16T12:22:55.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 39350K .......... .......... .......... .......... .......... 23% 24.7M 5s 2024-09-16T12:22:55.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 39400K .......... .......... .......... .......... .......... 23% 31.7M 5s 2024-09-16T12:22:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 39450K .......... .......... .......... .......... .......... 23% 49.9M 5s 2024-09-16T12:22:55.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 39500K .......... .......... .......... .......... .......... 23% 33.8M 5s 2024-09-16T12:22:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 39550K .......... .......... .......... .......... .......... 23% 21.7M 5s 2024-09-16T12:22:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 39600K .......... .......... .......... .......... .......... 23% 22.8M 5s 2024-09-16T12:22:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 39650K .......... .......... .......... .......... .......... 23% 33.1M 5s 2024-09-16T12:22:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 39700K .......... .......... .......... .......... .......... 23% 22.8M 5s 2024-09-16T12:22:55.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 39750K .......... .......... .......... .......... .......... 23% 51.3M 5s 2024-09-16T12:22:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 39800K .......... .......... .......... .......... .......... 23% 23.9M 5s 2024-09-16T12:22:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 39850K .......... .......... .......... .......... .......... 23% 53.1M 5s 2024-09-16T12:22:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 39900K .......... .......... .......... .......... .......... 23% 20.3M 5s 2024-09-16T12:22:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 39950K .......... .......... .......... .......... .......... 23% 22.1M 5s 2024-09-16T12:22:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 40000K .......... .......... .......... .......... .......... 23% 60.3M 5s 2024-09-16T12:22:55.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 40050K .......... .......... .......... .......... .......... 23% 21.9M 5s 2024-09-16T12:22:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 40100K .......... .......... .......... .......... .......... 23% 25.6M 5s 2024-09-16T12:22:55.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 40150K .......... .......... .......... .......... .......... 23% 45.8M 5s 2024-09-16T12:22:55.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 40200K .......... .......... .......... .......... .......... 23% 35.4M 5s 2024-09-16T12:22:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 40250K .......... .......... .......... .......... .......... 23% 24.1M 5s 2024-09-16T12:22:55.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 40300K .......... .......... .......... .......... .......... 23% 23.1M 5s 2024-09-16T12:22:55.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 40350K .......... .......... .......... .......... .......... 23% 26.1M 5s 2024-09-16T12:22:55.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 40400K .......... .......... .......... .......... .......... 23% 35.6M 5s 2024-09-16T12:22:55.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 40450K .......... .......... .......... .......... .......... 24% 31.4M 5s 2024-09-16T12:22:55.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 40500K .......... .......... .......... .......... .......... 24% 35.6M 5s 2024-09-16T12:22:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 40550K .......... .......... .......... .......... .......... 24% 26.3M 5s 2024-09-16T12:22:55.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 40600K .......... .......... .......... .......... .......... 24% 32.2M 5s 2024-09-16T12:22:55.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 40650K .......... .......... .......... .......... .......... 24% 24.1M 5s 2024-09-16T12:22:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 40700K .......... .......... .......... .......... .......... 24% 48.1M 5s 2024-09-16T12:22:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 40750K .......... .......... .......... .......... .......... 24% 24.1M 5s 2024-09-16T12:22:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 40800K .......... .......... .......... .......... .......... 24% 22.9M 5s 2024-09-16T12:22:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 40850K .......... .......... .......... .......... .......... 24% 38.4M 5s 2024-09-16T12:22:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 40900K .......... .......... .......... .......... .......... 24% 26.2M 5s 2024-09-16T12:22:55.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 40950K .......... .......... .......... .......... .......... 24% 22.6M 5s 2024-09-16T12:22:55.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 41000K .......... .......... .......... .......... .......... 24% 48.9M 5s 2024-09-16T12:22:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 41050K .......... .......... .......... .......... .......... 24% 24.5M 5s 2024-09-16T12:22:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 41100K .......... .......... .......... .......... .......... 24% 36.6M 5s 2024-09-16T12:22:55.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 41150K .......... .......... .......... .......... .......... 24% 27.0M 5s 2024-09-16T12:22:55.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 41200K .......... .......... .......... .......... .......... 24% 22.9M 5s 2024-09-16T12:22:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 41250K .......... .......... .......... .......... .......... 24% 42.5M 5s 2024-09-16T12:22:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 41300K .......... .......... .......... .......... .......... 24% 26.5M 5s 2024-09-16T12:22:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 41350K .......... .......... .......... .......... .......... 24% 21.5M 5s 2024-09-16T12:22:55.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 41400K .......... .......... .......... .......... .......... 24% 39.1M 5s 2024-09-16T12:22:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 41450K .......... .......... .......... .......... .......... 24% 26.3M 5s 2024-09-16T12:22:55.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 41500K .......... .......... .......... .......... .......... 24% 41.2M 5s 2024-09-16T12:22:55.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 41550K .......... .......... .......... .......... .......... 24% 24.0M 5s 2024-09-16T12:22:55.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 41600K .......... .......... .......... .......... .......... 24% 24.5M 5s 2024-09-16T12:22:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 41650K .......... .......... .......... .......... .......... 24% 35.5M 5s 2024-09-16T12:22:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 41700K .......... .......... .......... .......... .......... 24% 29.4M 5s 2024-09-16T12:22:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 41750K .......... .......... .......... .......... .......... 24% 41.9M 5s 2024-09-16T12:22:55.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 41800K .......... .......... .......... .......... .......... 24% 23.1M 5s 2024-09-16T12:22:55.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 41850K .......... .......... .......... .......... .......... 24% 36.6M 5s 2024-09-16T12:22:55.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 41900K .......... .......... .......... .......... .......... 24% 25.0M 5s 2024-09-16T12:22:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 41950K .......... .......... .......... .......... .......... 24% 21.5M 4s 2024-09-16T12:22:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 42000K .......... .......... .......... .......... .......... 24% 26.8M 4s 2024-09-16T12:22:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 42050K .......... .......... .......... .......... .......... 24% 54.5M 4s 2024-09-16T12:22:55.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 42100K .......... .......... .......... .......... .......... 25% 23.2M 4s 2024-09-16T12:22:55.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 42150K .......... .......... .......... .......... .......... 25% 35.9M 4s 2024-09-16T12:22:55.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 42200K .......... .......... .......... .......... .......... 25% 25.0M 4s 2024-09-16T12:22:55.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 42250K .......... .......... .......... .......... .......... 25% 54.2M 4s 2024-09-16T12:22:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 42300K .......... .......... .......... .......... .......... 25% 23.1M 4s 2024-09-16T12:22:55.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 42350K .......... .......... .......... .......... .......... 25% 27.1M 4s 2024-09-16T12:22:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 42400K .......... .......... .......... .......... .......... 25% 28.7M 4s 2024-09-16T12:22:55.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 42450K .......... .......... .......... .......... .......... 25% 21.0M 4s 2024-09-16T12:22:55.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 42500K .......... .......... .......... .......... .......... 25% 24.1M 4s 2024-09-16T12:22:55.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 42550K .......... .......... .......... .......... .......... 25% 33.2M 4s 2024-09-16T12:22:55.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 42600K .......... .......... .......... .......... .......... 25% 28.8M 4s 2024-09-16T12:22:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 42650K .......... .......... .......... .......... .......... 25% 27.2M 4s 2024-09-16T12:22:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 42700K .......... .......... .......... .......... .......... 25% 25.6M 4s 2024-09-16T12:22:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 42750K .......... .......... .......... .......... .......... 25% 25.6M 4s 2024-09-16T12:22:55.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 42800K .......... .......... .......... .......... .......... 25% 39.1M 4s 2024-09-16T12:22:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 42850K .......... .......... .......... .......... .......... 25% 26.0M 4s 2024-09-16T12:22:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 42900K .......... .......... .......... .......... .......... 25% 36.1M 4s 2024-09-16T12:22:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 42950K .......... .......... .......... .......... .......... 25% 24.1M 4s 2024-09-16T12:22:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 43000K .......... .......... .......... .......... .......... 25% 60.2M 4s 2024-09-16T12:22:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 43050K .......... .......... .......... .......... .......... 25% 21.6M 4s 2024-09-16T12:22:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 43100K .......... .......... .......... .......... .......... 25% 35.5M 4s 2024-09-16T12:22:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 43150K .......... .......... .......... .......... .......... 25% 22.7M 4s 2024-09-16T12:22:55.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 43200K .......... .......... .......... .......... .......... 25% 23.7M 4s 2024-09-16T12:22:55.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 43250K .......... .......... .......... .......... .......... 25% 26.2M 4s 2024-09-16T12:22:55.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 43300K .......... .......... .......... .......... .......... 25% 45.2M 4s 2024-09-16T12:22:55.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 43350K .......... .......... .......... .......... .......... 25% 22.8M 4s 2024-09-16T12:22:55.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 43400K .......... .......... .......... .......... .......... 25% 36.3M 4s 2024-09-16T12:22:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 43450K .......... .......... .......... .......... .......... 25% 25.3M 4s 2024-09-16T12:22:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 43500K .......... .......... .......... .......... .......... 25% 43.8M 4s 2024-09-16T12:22:55.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 43550K .......... .......... .......... .......... .......... 25% 26.7M 4s 2024-09-16T12:22:55.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 43600K .......... .......... .......... .......... .......... 25% 24.0M 4s 2024-09-16T12:22:55.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 43650K .......... .......... .......... .......... .......... 25% 28.9M 4s 2024-09-16T12:22:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 43700K .......... .......... .......... .......... .......... 25% 25.5M 4s 2024-09-16T12:22:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 43750K .......... .......... .......... .......... .......... 25% 23.8M 4s 2024-09-16T12:22:55.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 43800K .......... .......... .......... .......... .......... 26% 50.2M 4s 2024-09-16T12:22:55.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 43850K .......... .......... .......... .......... .......... 26% 25.4M 4s 2024-09-16T12:22:55.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 43900K .......... .......... .......... .......... .......... 26% 35.4M 4s 2024-09-16T12:22:55.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 43950K .......... .......... .......... .......... .......... 26% 24.7M 4s 2024-09-16T12:22:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 44000K .......... .......... .......... .......... .......... 26% 26.4M 4s 2024-09-16T12:22:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 44050K .......... .......... .......... .......... .......... 26% 44.2M 4s 2024-09-16T12:22:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 44100K .......... .......... .......... .......... .......... 26% 24.6M 4s 2024-09-16T12:22:55.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 44150K .......... .......... .......... .......... .......... 26% 36.4M 4s 2024-09-16T12:22:55.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 44200K .......... .......... .......... .......... .......... 26% 25.1M 4s 2024-09-16T12:22:55.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 44250K .......... .......... .......... .......... .......... 26% 22.5M 4s 2024-09-16T12:22:55.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 44300K .......... .......... .......... .......... .......... 26% 57.8M 4s 2024-09-16T12:22:55.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 44350K .......... .......... .......... .......... .......... 26% 21.8M 4s 2024-09-16T12:22:55.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 44400K .......... .......... .......... .......... .......... 26% 24.7M 4s 2024-09-16T12:22:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 44450K .......... .......... .......... .......... .......... 26% 36.9M 4s 2024-09-16T12:22:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 44500K .......... .......... .......... .......... .......... 26% 24.5M 4s 2024-09-16T12:22:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 44550K .......... .......... .......... .......... .......... 26% 41.6M 4s 2024-09-16T12:22:55.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 44600K .......... .......... .......... .......... .......... 26% 29.0M 4s 2024-09-16T12:22:55.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 44650K .......... .......... .......... .......... .......... 26% 33.2M 4s 2024-09-16T12:22:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 44700K .......... .......... .......... .......... .......... 26% 25.5M 4s 2024-09-16T12:22:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 44750K .......... .......... .......... .......... .......... 26% 22.9M 4s 2024-09-16T12:22:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 44800K .......... .......... .......... .......... .......... 26% 23.3M 4s 2024-09-16T12:22:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 44850K .......... .......... .......... .......... .......... 26% 52.8M 4s 2024-09-16T12:22:55.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 44900K .......... .......... .......... .......... .......... 26% 23.1M 4s 2024-09-16T12:22:55.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 44950K .......... .......... .......... .......... .......... 26% 35.4M 4s 2024-09-16T12:22:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 45000K .......... .......... .......... .......... .......... 26% 25.7M 4s 2024-09-16T12:22:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 45050K .......... .......... .......... .......... .......... 26% 45.5M 4s 2024-09-16T12:22:55.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 45100K .......... .......... .......... .......... .......... 26% 25.6M 4s 2024-09-16T12:22:55.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 45150K .......... .......... .......... .......... .......... 26% 23.2M 4s 2024-09-16T12:22:55.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 45200K .......... .......... .......... .......... .......... 26% 37.6M 4s 2024-09-16T12:22:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 45250K .......... .......... .......... .......... .......... 26% 24.9M 4s 2024-09-16T12:22:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 45300K .......... .......... .......... .......... .......... 26% 44.3M 4s 2024-09-16T12:22:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 45350K .......... .......... .......... .......... .......... 26% 25.5M 4s 2024-09-16T12:22:55.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 45400K .......... .......... .......... .......... .......... 26% 31.8M 4s 2024-09-16T12:22:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 45450K .......... .......... .......... .......... .......... 26% 24.2M 4s 2024-09-16T12:22:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 45500K .......... .......... .......... .......... .......... 27% 29.7M 4s 2024-09-16T12:22:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 45550K .......... .......... .......... .......... .......... 27% 22.8M 4s 2024-09-16T12:22:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 45600K .......... .......... .......... .......... .......... 27% 40.7M 4s 2024-09-16T12:22:55.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 45650K .......... .......... .......... .......... .......... 27% 24.8M 4s 2024-09-16T12:22:55.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 45700K .......... .......... .......... .......... .......... 27% 28.6M 4s 2024-09-16T12:22:55.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 45750K .......... .......... .......... .......... .......... 27% 26.1M 4s 2024-09-16T12:22:55.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 45800K .......... .......... .......... .......... .......... 27% 39.4M 4s 2024-09-16T12:22:55.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 45850K .......... .......... .......... .......... .......... 27% 25.5M 4s 2024-09-16T12:22:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 45900K .......... .......... .......... .......... .......... 27% 27.6M 4s 2024-09-16T12:22:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 45950K .......... .......... .......... .......... .......... 27% 21.4M 4s 2024-09-16T12:22:55.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 46000K .......... .......... .......... .......... .......... 27% 26.1M 4s 2024-09-16T12:22:55.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 46050K .......... .......... .......... .......... .......... 27% 52.9M 4s 2024-09-16T12:22:55.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 46100K .......... .......... .......... .......... .......... 27% 26.1M 4s 2024-09-16T12:22:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 46150K .......... .......... .......... .......... .......... 27% 26.0M 4s 2024-09-16T12:22:55.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 46200K .......... .......... .......... .......... .......... 27% 29.2M 4s 2024-09-16T12:22:55.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 46250K .......... .......... .......... .......... .......... 27% 25.6M 4s 2024-09-16T12:22:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 46300K .......... .......... .......... .......... .......... 27% 45.6M 4s 2024-09-16T12:22:55.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 46350K .......... .......... .......... .......... .......... 27% 24.1M 4s 2024-09-16T12:22:55.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 46400K .......... .......... .......... .......... .......... 27% 25.0M 4s 2024-09-16T12:22:55.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 46450K .......... .......... .......... .......... .......... 27% 36.6M 4s 2024-09-16T12:22:55.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 46500K .......... .......... .......... .......... .......... 27% 22.7M 4s 2024-09-16T12:22:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 46550K .......... .......... .......... .......... .......... 27% 24.8M 4s 2024-09-16T12:22:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 46600K .......... .......... .......... .......... .......... 27% 58.7M 4s 2024-09-16T12:22:55.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 46650K .......... .......... .......... .......... .......... 27% 22.3M 4s 2024-09-16T12:22:55.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 46700K .......... .......... .......... .......... .......... 27% 38.9M 4s 2024-09-16T12:22:55.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 46750K .......... .......... .......... .......... .......... 27% 22.6M 4s 2024-09-16T12:22:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 46800K .......... .......... .......... .......... .......... 27% 24.3M 4s 2024-09-16T12:22:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 46850K .......... .......... .......... .......... .......... 27% 44.6M 4s 2024-09-16T12:22:55.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 46900K .......... .......... .......... .......... .......... 27% 24.9M 4s 2024-09-16T12:22:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 46950K .......... .......... .......... .......... .......... 27% 24.8M 4s 2024-09-16T12:22:55.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 47000K .......... .......... .......... .......... .......... 27% 33.7M 4s 2024-09-16T12:22:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 47050K .......... .......... .......... .......... .......... 27% 23.2M 4s 2024-09-16T12:22:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 47100K .......... .......... .......... .......... .......... 27% 53.3M 4s 2024-09-16T12:22:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 47150K .......... .......... .......... .......... .......... 28% 23.0M 4s 2024-09-16T12:22:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 47200K .......... .......... .......... .......... .......... 28% 24.6M 4s 2024-09-16T12:22:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 47250K .......... .......... .......... .......... .......... 28% 37.1M 4s 2024-09-16T12:22:55.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 47300K .......... .......... .......... .......... .......... 28% 29.9M 4s 2024-09-16T12:22:55.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 47350K .......... .......... .......... .......... .......... 28% 37.9M 4s 2024-09-16T12:22:55.360 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 47400K .......... .......... .......... .......... .......... 28% 23.3M 4s 2024-09-16T12:22:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 47450K .......... .......... .......... .......... .......... 28% 36.2M 4s 2024-09-16T12:22:55.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 47500K .......... .......... .......... .......... .......... 28% 29.7M 4s 2024-09-16T12:22:55.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 47550K .......... .......... .......... .......... .......... 28% 21.4M 4s 2024-09-16T12:22:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 47600K .......... .......... .......... .......... .......... 28% 22.7M 4s 2024-09-16T12:22:55.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 47650K .......... .......... .......... .......... .......... 28% 50.6M 4s 2024-09-16T12:22:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 47700K .......... .......... .......... .......... .......... 28% 23.8M 4s 2024-09-16T12:22:55.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 47750K .......... .......... .......... .......... .......... 28% 36.9M 4s 2024-09-16T12:22:55.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 47800K .......... .......... .......... .......... .......... 28% 25.5M 4s 2024-09-16T12:22:55.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 47850K .......... .......... .......... .......... .......... 28% 41.1M 4s 2024-09-16T12:22:55.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 47900K .......... .......... .......... .......... .......... 28% 26.6M 4s 2024-09-16T12:22:55.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 47950K .......... .......... .......... .......... .......... 28% 22.4M 4s 2024-09-16T12:22:55.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 48000K .......... .......... .......... .......... .......... 28% 35.0M 4s 2024-09-16T12:22:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 48050K .......... .......... .......... .......... .......... 28% 25.3M 4s 2024-09-16T12:22:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 48100K .......... .......... .......... .......... .......... 28% 43.7M 4s 2024-09-16T12:22:55.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 48150K .......... .......... .......... .......... .......... 28% 26.3M 4s 2024-09-16T12:22:55.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 48200K .......... .......... .......... .......... .......... 28% 32.0M 4s 2024-09-16T12:22:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 48250K .......... .......... .......... .......... .......... 28% 24.8M 4s 2024-09-16T12:22:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 48300K .......... .......... .......... .......... .......... 28% 25.8M 4s 2024-09-16T12:22:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 48350K .......... .......... .......... .......... .......... 28% 24.0M 4s 2024-09-16T12:22:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 48400K .......... .......... .......... .......... .......... 28% 44.6M 4s 2024-09-16T12:22:55.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 48450K .......... .......... .......... .......... .......... 28% 25.6M 4s 2024-09-16T12:22:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 48500K .......... .......... .......... .......... .......... 28% 37.3M 4s 2024-09-16T12:22:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 48550K .......... .......... .......... .......... .......... 28% 24.3M 4s 2024-09-16T12:22:55.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 48600K .......... .......... .......... .......... .......... 28% 45.6M 4s 2024-09-16T12:22:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 48650K .......... .......... .......... .......... .......... 28% 25.9M 4s 2024-09-16T12:22:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 48700K .......... .......... .......... .......... .......... 28% 34.9M 4s 2024-09-16T12:22:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 48750K .......... .......... .......... .......... .......... 28% 21.3M 4s 2024-09-16T12:22:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 48800K .......... .......... .......... .......... .......... 28% 26.8M 4s 2024-09-16T12:22:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 48850K .......... .......... .......... .......... .......... 29% 27.5M 4s 2024-09-16T12:22:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 48900K .......... .......... .......... .......... .......... 29% 38.3M 4s 2024-09-16T12:22:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 48950K .......... .......... .......... .......... .......... 29% 28.0M 4s 2024-09-16T12:22:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 49000K .......... .......... .......... .......... .......... 29% 24.3M 4s 2024-09-16T12:22:55.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 49050K .......... .......... .......... .......... .......... 29% 27.5M 4s 2024-09-16T12:22:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 49100K .......... .......... .......... .......... .......... 29% 42.6M 4s 2024-09-16T12:22:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 49150K .......... .......... .......... .......... .......... 29% 25.8M 4s 2024-09-16T12:22:55.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 49200K .......... .......... .......... .......... .......... 29% 24.0M 4s 2024-09-16T12:22:55.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 49250K .......... .......... .......... .......... .......... 29% 26.7M 4s 2024-09-16T12:22:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 49300K .......... .......... .......... .......... .......... 29% 23.4M 4s 2024-09-16T12:22:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 49350K .......... .......... .......... .......... .......... 29% 25.9M 4s 2024-09-16T12:22:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 49400K .......... .......... .......... .......... .......... 29% 53.4M 4s 2024-09-16T12:22:55.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 49450K .......... .......... .......... .......... .......... 29% 22.8M 4s 2024-09-16T12:22:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 49500K .......... .......... .......... .......... .......... 29% 37.2M 4s 2024-09-16T12:22:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 49550K .......... .......... .......... .......... .......... 29% 24.7M 4s 2024-09-16T12:22:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 49600K .......... .......... .......... .......... .......... 29% 24.0M 4s 2024-09-16T12:22:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 49650K .......... .......... .......... .......... .......... 29% 47.0M 4s 2024-09-16T12:22:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 49700K .......... .......... .......... .......... .......... 29% 25.5M 4s 2024-09-16T12:22:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 49750K .......... .......... .......... .......... .......... 29% 33.7M 4s 2024-09-16T12:22:55.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 49800K .......... .......... .......... .......... .......... 29% 25.6M 4s 2024-09-16T12:22:55.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 49850K .......... .......... .......... .......... .......... 29% 47.3M 4s 2024-09-16T12:22:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 49900K .......... .......... .......... .......... .......... 29% 25.0M 4s 2024-09-16T12:22:55.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 49950K .......... .......... .......... .......... .......... 29% 23.4M 4s 2024-09-16T12:22:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50000K .......... .......... .......... .......... .......... 29% 24.4M 4s 2024-09-16T12:22:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50050K .......... .......... .......... .......... .......... 29% 36.1M 4s 2024-09-16T12:22:55.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50100K .......... .......... .......... .......... .......... 29% 24.4M 4s 2024-09-16T12:22:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50150K .......... .......... .......... .......... .......... 29% 39.2M 4s 2024-09-16T12:22:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50200K .......... .......... .......... .......... .......... 29% 26.4M 4s 2024-09-16T12:22:55.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50250K .......... .......... .......... .......... .......... 29% 37.1M 4s 2024-09-16T12:22:55.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50300K .......... .......... .......... .......... .......... 29% 23.3M 4s 2024-09-16T12:22:55.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50350K .......... .......... .......... .......... .......... 29% 24.9M 4s 2024-09-16T12:22:55.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50400K .......... .......... .......... .......... .......... 29% 41.8M 4s 2024-09-16T12:22:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50450K .......... .......... .......... .......... .......... 29% 26.5M 4s 2024-09-16T12:22:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50500K .......... .......... .......... .......... .......... 29% 32.6M 4s 2024-09-16T12:22:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50550K .......... .......... .......... .......... .......... 30% 23.3M 4s 2024-09-16T12:22:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50600K .......... .......... .......... .......... .......... 30% 26.9M 4s 2024-09-16T12:22:55.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50650K .......... .......... .......... .......... .......... 30% 38.5M 4s 2024-09-16T12:22:55.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50700K .......... .......... .......... .......... .......... 30% 25.8M 4s 2024-09-16T12:22:55.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50750K .......... .......... .......... .......... .......... 30% 24.6M 4s 2024-09-16T12:22:55.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50800K .......... .......... .......... .......... .......... 30% 36.1M 4s 2024-09-16T12:22:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50850K .......... .......... .......... .......... .......... 30% 25.8M 4s 2024-09-16T12:22:55.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50900K .......... .......... .......... .......... .......... 30% 43.6M 4s 2024-09-16T12:22:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 50950K .......... .......... .......... .......... .......... 30% 24.5M 4s 2024-09-16T12:22:55.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 51000K .......... .......... .......... .......... .......... 30% 36.5M 4s 2024-09-16T12:22:55.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 51050K .......... .......... .......... .......... .......... 30% 22.9M 4s 2024-09-16T12:22:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 51100K .......... .......... .......... .......... .......... 30% 26.1M 4s 2024-09-16T12:22:55.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 51150K .......... .......... .......... .......... .......... 30% 25.3M 4s 2024-09-16T12:22:55.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 51200K .......... .......... .......... .......... .......... 30% 44.1M 4s 2024-09-16T12:22:55.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 51250K .......... .......... .......... .......... .......... 30% 24.6M 4s 2024-09-16T12:22:55.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 51300K .......... .......... .......... .......... .......... 30% 37.7M 4s 2024-09-16T12:22:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 51350K .......... .......... .......... .......... .......... 30% 24.2M 4s 2024-09-16T12:22:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 51400K .......... .......... .......... .......... .......... 30% 43.7M 4s 2024-09-16T12:22:55.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 51450K .......... .......... .......... .......... .......... 30% 28.8M 4s 2024-09-16T12:22:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 51500K .......... .......... .......... .......... .......... 30% 33.9M 4s 2024-09-16T12:22:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 51550K .......... .......... .......... .......... .......... 30% 23.1M 4s 2024-09-16T12:22:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 51600K .......... .......... .......... .......... .......... 30% 24.3M 4s 2024-09-16T12:22:55.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 51650K .......... .......... .......... .......... .......... 30% 24.0M 4s 2024-09-16T12:22:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 51700K .......... .......... .......... .......... .......... 30% 47.5M 4s 2024-09-16T12:22:55.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 51750K .......... .......... .......... .......... .......... 30% 23.8M 4s 2024-09-16T12:22:55.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 51800K .......... .......... .......... .......... .......... 30% 38.0M 4s 2024-09-16T12:22:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 51850K .......... .......... .......... .......... .......... 30% 25.2M 4s 2024-09-16T12:22:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 51900K .......... .......... .......... .......... .......... 30% 47.6M 4s 2024-09-16T12:22:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 51950K .......... .......... .......... .......... .......... 30% 23.0M 4s 2024-09-16T12:22:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 52000K .......... .......... .......... .......... .......... 30% 25.6M 4s 2024-09-16T12:22:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 52050K .......... .......... .......... .......... .......... 30% 38.2M 4s 2024-09-16T12:22:55.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 52100K .......... .......... .......... .......... .......... 30% 24.6M 4s 2024-09-16T12:22:55.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 52150K .......... .......... .......... .......... .......... 30% 23.0M 4s 2024-09-16T12:22:55.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 52200K .......... .......... .......... .......... .......... 30% 50.1M 4s 2024-09-16T12:22:55.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 52250K .......... .......... .......... .......... .......... 31% 23.7M 4s 2024-09-16T12:22:55.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 52300K .......... .......... .......... .......... .......... 31% 36.6M 4s 2024-09-16T12:22:55.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 52350K .......... .......... .......... .......... .......... 31% 22.5M 4s 2024-09-16T12:22:55.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 52400K .......... .......... .......... .......... .......... 31% 26.0M 4s 2024-09-16T12:22:55.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 52450K .......... .......... .......... .......... .......... 31% 41.5M 4s 2024-09-16T12:22:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 52500K .......... .......... .......... .......... .......... 31% 24.6M 4s 2024-09-16T12:22:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 52550K .......... .......... .......... .......... .......... 31% 37.0M 4s 2024-09-16T12:22:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 52600K .......... .......... .......... .......... .......... 31% 24.5M 4s 2024-09-16T12:22:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 52650K .......... .......... .......... .......... .......... 31% 51.1M 4s 2024-09-16T12:22:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 52700K .......... .......... .......... .......... .......... 31% 24.0M 4s 2024-09-16T12:22:55.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 52750K .......... .......... .......... .......... .......... 31% 22.7M 4s 2024-09-16T12:22:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 52800K .......... .......... .......... .......... .......... 31% 31.8M 4s 2024-09-16T12:22:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 52850K .......... .......... .......... .......... .......... 31% 22.6M 4s 2024-09-16T12:22:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 52900K .......... .......... .......... .......... .......... 31% 30.3M 4s 2024-09-16T12:22:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 52950K .......... .......... .......... .......... .......... 31% 43.7M 4s 2024-09-16T12:22:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 53000K .......... .......... .......... .......... .......... 31% 24.6M 4s 2024-09-16T12:22:55.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 53050K .......... .......... .......... .......... .......... 31% 41.0M 4s 2024-09-16T12:22:55.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 53100K .......... .......... .......... .......... .......... 31% 29.0M 4s 2024-09-16T12:22:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 53150K .......... .......... .......... .......... .......... 31% 21.4M 4s 2024-09-16T12:22:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 53200K .......... .......... .......... .......... .......... 31% 40.2M 4s 2024-09-16T12:22:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 53250K .......... .......... .......... .......... .......... 31% 24.6M 4s 2024-09-16T12:22:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 53300K .......... .......... .......... .......... .......... 31% 37.4M 4s 2024-09-16T12:22:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 53350K .......... .......... .......... .......... .......... 31% 24.9M 4s 2024-09-16T12:22:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 53400K .......... .......... .......... .......... .......... 31% 23.7M 4s 2024-09-16T12:22:55.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 53450K .......... .......... .......... .......... .......... 31% 46.2M 4s 2024-09-16T12:22:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 53500K .......... .......... .......... .......... .......... 31% 25.0M 4s 2024-09-16T12:22:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 53550K .......... .......... .......... .......... .......... 31% 24.1M 4s 2024-09-16T12:22:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 53600K .......... .......... .......... .......... .......... 31% 36.3M 4s 2024-09-16T12:22:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 53650K .......... .......... .......... .......... .......... 31% 25.1M 4s 2024-09-16T12:22:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 53700K .......... .......... .......... .......... .......... 31% 48.0M 4s 2024-09-16T12:22:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 53750K .......... .......... .......... .......... .......... 31% 23.9M 4s 2024-09-16T12:22:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 53800K .......... .......... .......... .......... .......... 31% 38.2M 4s 2024-09-16T12:22:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 53850K .......... .......... .......... .......... .......... 31% 23.3M 4s 2024-09-16T12:22:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 53900K .......... .......... .......... .......... .......... 32% 27.2M 4s 2024-09-16T12:22:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 53950K .......... .......... .......... .......... .......... 32% 21.3M 4s 2024-09-16T12:22:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 54000K .......... .......... .......... .......... .......... 32% 50.7M 4s 2024-09-16T12:22:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 54050K .......... .......... .......... .......... .......... 32% 24.1M 4s 2024-09-16T12:22:55.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 54100K .......... .......... .......... .......... .......... 32% 37.2M 4s 2024-09-16T12:22:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 54150K .......... .......... .......... .......... .......... 32% 24.0M 4s 2024-09-16T12:22:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 54200K .......... .......... .......... .......... .......... 32% 46.5M 4s 2024-09-16T12:22:55.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 54250K .......... .......... .......... .......... .......... 32% 24.9M 4s 2024-09-16T12:22:55.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 54300K .......... .......... .......... .......... .......... 32% 40.7M 4s 2024-09-16T12:22:55.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 54350K .......... .......... .......... .......... .......... 32% 22.7M 4s 2024-09-16T12:22:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 54400K .......... .......... .......... .......... .......... 32% 23.0M 4s 2024-09-16T12:22:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 54450K .......... .......... .......... .......... .......... 32% 24.3M 4s 2024-09-16T12:22:55.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 54500K .......... .......... .......... .......... .......... 32% 54.5M 4s 2024-09-16T12:22:55.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 54550K .......... .......... .......... .......... .......... 32% 23.1M 4s 2024-09-16T12:22:55.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 54600K .......... .......... .......... .......... .......... 32% 38.2M 4s 2024-09-16T12:22:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 54650K .......... .......... .......... .......... .......... 32% 23.8M 4s 2024-09-16T12:22:55.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 54700K .......... .......... .......... .......... .......... 32% 45.2M 4s 2024-09-16T12:22:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 54750K .......... .......... .......... .......... .......... 32% 26.5M 4s 2024-09-16T12:22:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 54800K .......... .......... .......... .......... .......... 32% 23.2M 4s 2024-09-16T12:22:55.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 54850K .......... .......... .......... .......... .......... 32% 36.9M 4s 2024-09-16T12:22:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 54900K .......... .......... .......... .......... .......... 32% 25.2M 4s 2024-09-16T12:22:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 54950K .......... .......... .......... .......... .......... 32% 22.0M 4s 2024-09-16T12:22:55.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 55000K .......... .......... .......... .......... .......... 32% 55.9M 4s 2024-09-16T12:22:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 55050K .......... .......... .......... .......... .......... 32% 33.8M 4s 2024-09-16T12:22:55.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 55100K .......... .......... .......... .......... .......... 32% 25.2M 4s 2024-09-16T12:22:55.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 55150K .......... .......... .......... .......... .......... 32% 24.7M 4s 2024-09-16T12:22:55.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 55200K .......... .......... .......... .......... .......... 32% 32.0M 4s 2024-09-16T12:22:55.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 55250K .......... .......... .......... .......... .......... 32% 33.1M 4s 2024-09-16T12:22:55.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 55300K .......... .......... .......... .......... .......... 32% 24.2M 4s 2024-09-16T12:22:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 55350K .......... .......... .......... .......... .......... 32% 36.1M 4s 2024-09-16T12:22:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 55400K .......... .......... .......... .......... .......... 32% 25.2M 4s 2024-09-16T12:22:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 55450K .......... .......... .......... .......... .......... 32% 45.9M 4s 2024-09-16T12:22:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 55500K .......... .......... .......... .......... .......... 32% 23.9M 4s 2024-09-16T12:22:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 55550K .......... .......... .......... .......... .......... 32% 24.5M 4s 2024-09-16T12:22:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 55600K .......... .......... .......... .......... .......... 33% 32.3M 4s 2024-09-16T12:22:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 55650K .......... .......... .......... .......... .......... 33% 25.4M 4s 2024-09-16T12:22:55.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 55700K .......... .......... .......... .......... .......... 33% 25.6M 4s 2024-09-16T12:22:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 55750K .......... .......... .......... .......... .......... 33% 46.3M 4s 2024-09-16T12:22:55.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 55800K .......... .......... .......... .......... .......... 33% 23.8M 4s 2024-09-16T12:22:55.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 55850K .......... .......... .......... .......... .......... 33% 39.9M 4s 2024-09-16T12:22:55.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 55900K .......... .......... .......... .......... .......... 33% 24.0M 4s 2024-09-16T12:22:55.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 55950K .......... .......... .......... .......... .......... 33% 23.6M 4s 2024-09-16T12:22:55.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 56000K .......... .......... .......... .......... .......... 33% 45.1M 4s 2024-09-16T12:22:55.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 56050K .......... .......... .......... .......... .......... 33% 26.0M 4s 2024-09-16T12:22:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 56100K .......... .......... .......... .......... .......... 33% 35.9M 4s 2024-09-16T12:22:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 56150K .......... .......... .......... .......... .......... 33% 26.2M 4s 2024-09-16T12:22:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 56200K .......... .......... .......... .......... .......... 33% 21.7M 4s 2024-09-16T12:22:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 56250K .......... .......... .......... .......... .......... 33% 45.8M 4s 2024-09-16T12:22:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 56300K .......... .......... .......... .......... .......... 33% 25.3M 4s 2024-09-16T12:22:55.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 56350K .......... .......... .......... .......... .......... 33% 27.5M 4s 2024-09-16T12:22:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 56400K .......... .......... .......... .......... .......... 33% 33.9M 4s 2024-09-16T12:22:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 56450K .......... .......... .......... .......... .......... 33% 22.9M 4s 2024-09-16T12:22:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 56500K .......... .......... .......... .......... .......... 33% 48.1M 4s 2024-09-16T12:22:55.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 56550K .......... .......... .......... .......... .......... 33% 24.7M 4s 2024-09-16T12:22:55.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 56600K .......... .......... .......... .......... .......... 33% 34.8M 4s 2024-09-16T12:22:55.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 56650K .......... .......... .......... .......... .......... 33% 24.8M 4s 2024-09-16T12:22:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 56700K .......... .......... .......... .......... .......... 33% 23.2M 4s 2024-09-16T12:22:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 56750K .......... .......... .......... .......... .......... 33% 24.8M 4s 2024-09-16T12:22:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 56800K .......... .......... .......... .......... .......... 33% 54.9M 4s 2024-09-16T12:22:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 56850K .......... .......... .......... .......... .......... 33% 25.8M 4s 2024-09-16T12:22:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 56900K .......... .......... .......... .......... .......... 33% 33.9M 4s 2024-09-16T12:22:55.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 56950K .......... .......... .......... .......... .......... 33% 22.4M 4s 2024-09-16T12:22:55.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 57000K .......... .......... .......... .......... .......... 33% 45.8M 4s 2024-09-16T12:22:55.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 57050K .......... .......... .......... .......... .......... 33% 25.6M 4s 2024-09-16T12:22:55.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 57100K .......... .......... .......... .......... .......... 33% 37.3M 4s 2024-09-16T12:22:55.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 57150K .......... .......... .......... .......... .......... 33% 23.9M 4s 2024-09-16T12:22:55.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 57200K .......... .......... .......... .......... .......... 33% 25.2M 4s 2024-09-16T12:22:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 57250K .......... .......... .......... .......... .......... 33% 22.9M 4s 2024-09-16T12:22:55.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 57300K .......... .......... .......... .......... .......... 34% 57.0M 4s 2024-09-16T12:22:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 57350K .......... .......... .......... .......... .......... 34% 31.4M 4s 2024-09-16T12:22:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 57400K .......... .......... .......... .......... .......... 34% 25.3M 4s 2024-09-16T12:22:55.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 57450K .......... .......... .......... .......... .......... 34% 25.4M 4s 2024-09-16T12:22:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 57500K .......... .......... .......... .......... .......... 34% 49.6M 4s 2024-09-16T12:22:55.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 57550K .......... .......... .......... .......... .......... 34% 23.7M 4s 2024-09-16T12:22:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 57600K .......... .......... .......... .......... .......... 34% 23.4M 4s 2024-09-16T12:22:55.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 57650K .......... .......... .......... .......... .......... 34% 36.8M 4s 2024-09-16T12:22:55.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 57700K .......... .......... .......... .......... .......... 34% 38.9M 4s 2024-09-16T12:22:55.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 57750K .......... .......... .......... .......... .......... 34% 30.3M 4s 2024-09-16T12:22:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 57800K .......... .......... .......... .......... .......... 34% 22.4M 4s 2024-09-16T12:22:55.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 57850K .......... .......... .......... .......... .......... 34% 36.4M 4s 2024-09-16T12:22:55.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 57900K .......... .......... .......... .......... .......... 34% 24.7M 4s 2024-09-16T12:22:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 57950K .......... .......... .......... .......... .......... 34% 24.5M 4s 2024-09-16T12:22:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 58000K .......... .......... .......... .......... .......... 34% 23.8M 4s 2024-09-16T12:22:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 58050K .......... .......... .......... .......... .......... 34% 46.7M 4s 2024-09-16T12:22:55.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 58100K .......... .......... .......... .......... .......... 34% 25.3M 4s 2024-09-16T12:22:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 58150K .......... .......... .......... .......... .......... 34% 35.9M 4s 2024-09-16T12:22:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 58200K .......... .......... .......... .......... .......... 34% 24.7M 4s 2024-09-16T12:22:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 58250K .......... .......... .......... .......... .......... 34% 42.2M 4s 2024-09-16T12:22:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 58300K .......... .......... .......... .......... .......... 34% 25.3M 4s 2024-09-16T12:22:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 58350K .......... .......... .......... .......... .......... 34% 31.6M 4s 2024-09-16T12:22:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 58400K .......... .......... .......... .......... .......... 34% 23.6M 4s 2024-09-16T12:22:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 58450K .......... .......... .......... .......... .......... 34% 25.5M 4s 2024-09-16T12:22:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 58500K .......... .......... .......... .......... .......... 34% 26.3M 4s 2024-09-16T12:22:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 58550K .......... .......... .......... .......... .......... 34% 56.5M 4s 2024-09-16T12:22:55.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 58600K .......... .......... .......... .......... .......... 34% 22.1M 4s 2024-09-16T12:22:55.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 58650K .......... .......... .......... .......... .......... 34% 35.3M 4s 2024-09-16T12:22:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 58700K .......... .......... .......... .......... .......... 34% 25.9M 4s 2024-09-16T12:22:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 58750K .......... .......... .......... .......... .......... 34% 24.5M 4s 2024-09-16T12:22:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 58800K .......... .......... .......... .......... .......... 34% 44.3M 4s 2024-09-16T12:22:55.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 58850K .......... .......... .......... .......... .......... 34% 25.6M 4s 2024-09-16T12:22:55.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 58900K .......... .......... .......... .......... .......... 34% 33.5M 4s 2024-09-16T12:22:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 58950K .......... .......... .......... .......... .......... 35% 25.1M 4s 2024-09-16T12:22:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 59000K .......... .......... .......... .......... .......... 35% 24.5M 4s 2024-09-16T12:22:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 59050K .......... .......... .......... .......... .......... 35% 53.0M 4s 2024-09-16T12:22:55.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 59100K .......... .......... .......... .......... .......... 35% 23.6M 4s 2024-09-16T12:22:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 59150K .......... .......... .......... .......... .......... 35% 23.4M 4s 2024-09-16T12:22:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 59200K .......... .......... .......... .......... .......... 35% 36.4M 4s 2024-09-16T12:22:55.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 59250K .......... .......... .......... .......... .......... 35% 24.9M 4s 2024-09-16T12:22:55.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 59300K .......... .......... .......... .......... .......... 35% 47.0M 4s 2024-09-16T12:22:55.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 59350K .......... .......... .......... .......... .......... 35% 24.7M 4s 2024-09-16T12:22:55.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 59400K .......... .......... .......... .......... .......... 35% 26.8M 4s 2024-09-16T12:22:55.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 59450K .......... .......... .......... .......... .......... 35% 25.4M 4s 2024-09-16T12:22:55.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 59500K .......... .......... .......... .......... .......... 35% 23.0M 4s 2024-09-16T12:22:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 59550K .......... .......... .......... .......... .......... 35% 24.9M 4s 2024-09-16T12:22:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 59600K .......... .......... .......... .......... .......... 35% 51.6M 4s 2024-09-16T12:22:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 59650K .......... .......... .......... .......... .......... 35% 19.4M 4s 2024-09-16T12:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 59700K .......... .......... .......... .......... .......... 35% 34.9M 4s 2024-09-16T12:22:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 59750K .......... .......... .......... .......... .......... 35% 24.0M 4s 2024-09-16T12:22:55.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 59800K .......... .......... .......... .......... .......... 35% 47.6M 4s 2024-09-16T12:22:55.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 59850K .......... .......... .......... .......... .......... 35% 25.6M 4s 2024-09-16T12:22:55.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 59900K .......... .......... .......... .......... .......... 35% 36.7M 4s 2024-09-16T12:22:55.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 59950K .......... .......... .......... .......... .......... 35% 23.9M 4s 2024-09-16T12:22:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 60000K .......... .......... .......... .......... .......... 35% 25.4M 4s 2024-09-16T12:22:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 60050K .......... .......... .......... .......... .......... 35% 23.2M 4s 2024-09-16T12:22:55.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 60100K .......... .......... .......... .......... .......... 35% 55.6M 4s 2024-09-16T12:22:55.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 60150K .......... .......... .......... .......... .......... 35% 32.0M 4s 2024-09-16T12:22:55.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 60200K .......... .......... .......... .......... .......... 35% 24.5M 4s 2024-09-16T12:22:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 60250K .......... .......... .......... .......... .......... 35% 24.4M 4s 2024-09-16T12:22:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 60300K .......... .......... .......... .......... .......... 35% 49.8M 4s 2024-09-16T12:22:55.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 60350K .......... .......... .......... .......... .......... 35% 25.1M 4s 2024-09-16T12:22:55.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 60400K .......... .......... .......... .......... .......... 35% 24.8M 4s 2024-09-16T12:22:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 60450K .......... .......... .......... .......... .......... 35% 36.5M 4s 2024-09-16T12:22:55.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 60500K .......... .......... .......... .......... .......... 35% 24.6M 4s 2024-09-16T12:22:55.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 60550K .......... .......... .......... .......... .......... 35% 42.9M 4s 2024-09-16T12:22:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 60600K .......... .......... .......... .......... .......... 35% 24.9M 4s 2024-09-16T12:22:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 60650K .......... .......... .......... .......... .......... 36% 34.7M 4s 2024-09-16T12:22:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 60700K .......... .......... .......... .......... .......... 36% 24.7M 4s 2024-09-16T12:22:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 60750K .......... .......... .......... .......... .......... 36% 24.0M 4s 2024-09-16T12:22:55.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 60800K .......... .......... .......... .......... .......... 36% 29.9M 4s 2024-09-16T12:22:55.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 60850K .......... .......... .......... .......... .......... 36% 35.7M 4s 2024-09-16T12:22:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 60900K .......... .......... .......... .......... .......... 36% 25.9M 4s 2024-09-16T12:22:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 60950K .......... .......... .......... .......... .......... 36% 34.5M 4s 2024-09-16T12:22:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 61000K .......... .......... .......... .......... .......... 36% 25.2M 4s 2024-09-16T12:22:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 61050K .......... .......... .......... .......... .......... 36% 46.2M 4s 2024-09-16T12:22:55.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 61100K .......... .......... .......... .......... .......... 36% 24.7M 4s 2024-09-16T12:22:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 61150K .......... .......... .......... .......... .......... 36% 24.7M 4s 2024-09-16T12:22:55.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 61200K .......... .......... .......... .......... .......... 36% 37.4M 4s 2024-09-16T12:22:55.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 61250K .......... .......... .......... .......... .......... 36% 22.4M 4s 2024-09-16T12:22:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 61300K .......... .......... .......... .......... .......... 36% 26.6M 4s 2024-09-16T12:22:55.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 61350K .......... .......... .......... .......... .......... 36% 39.4M 4s 2024-09-16T12:22:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 61400K .......... .......... .......... .......... .......... 36% 25.5M 4s 2024-09-16T12:22:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 61450K .......... .......... .......... .......... .......... 36% 39.2M 4s 2024-09-16T12:22:55.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 61500K .......... .......... .......... .......... .......... 36% 24.6M 4s 2024-09-16T12:22:55.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 61550K .......... .......... .......... .......... .......... 36% 23.0M 4s 2024-09-16T12:22:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 61600K .......... .......... .......... .......... .......... 36% 43.9M 4s 2024-09-16T12:22:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 61650K .......... .......... .......... .......... .......... 36% 25.4M 4s 2024-09-16T12:22:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 61700K .......... .......... .......... .......... .......... 36% 36.9M 4s 2024-09-16T12:22:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 61750K .......... .......... .......... .......... .......... 36% 24.1M 4s 2024-09-16T12:22:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 61800K .......... .......... .......... .......... .......... 36% 23.9M 4s 2024-09-16T12:22:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 61850K .......... .......... .......... .......... .......... 36% 50.2M 4s 2024-09-16T12:22:55.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 61900K .......... .......... .......... .......... .......... 36% 28.2M 4s 2024-09-16T12:22:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 61950K .......... .......... .......... .......... .......... 36% 25.7M 4s 2024-09-16T12:22:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 62000K .......... .......... .......... .......... .......... 36% 26.7M 4s 2024-09-16T12:22:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 62050K .......... .......... .......... .......... .......... 36% 25.8M 4s 2024-09-16T12:22:55.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 62100K .......... .......... .......... .......... .......... 36% 43.5M 4s 2024-09-16T12:22:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 62150K .......... .......... .......... .......... .......... 36% 26.9M 4s 2024-09-16T12:22:55.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 62200K .......... .......... .......... .......... .......... 36% 34.4M 4s 2024-09-16T12:22:55.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 62250K .......... .......... .......... .......... .......... 36% 24.7M 4s 2024-09-16T12:22:55.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 62300K .......... .......... .......... .......... .......... 36% 43.5M 4s 2024-09-16T12:22:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 62350K .......... .......... .......... .......... .......... 37% 17.4M 4s 2024-09-16T12:22:55.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 62400K .......... .......... .......... .......... .......... 37% 47.6M 4s 2024-09-16T12:22:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 62450K .......... .......... .......... .......... .......... 37% 22.7M 4s 2024-09-16T12:22:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 62500K .......... .......... .......... .......... .......... 37% 34.3M 4s 2024-09-16T12:22:55.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 62550K .......... .......... .......... .......... .......... 37% 24.5M 4s 2024-09-16T12:22:55.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 62600K .......... .......... .......... .......... .......... 37% 48.1M 4s 2024-09-16T12:22:55.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 62650K .......... .......... .......... .......... .......... 37% 25.7M 4s 2024-09-16T12:22:55.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 62700K .......... .......... .......... .......... .......... 37% 35.8M 4s 2024-09-16T12:22:55.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 62750K .......... .......... .......... .......... .......... 37% 14.9M 4s 2024-09-16T12:22:55.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 62800K .......... .......... .......... .......... .......... 37% 43.5M 4s 2024-09-16T12:22:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 62850K .......... .......... .......... .......... .......... 37% 39.7M 4s 2024-09-16T12:22:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 62900K .......... .......... .......... .......... .......... 37% 24.6M 4s 2024-09-16T12:22:55.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 62950K .......... .......... .......... .......... .......... 37% 32.6M 4s 2024-09-16T12:22:55.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 63000K .......... .......... .......... .......... .......... 37% 23.8M 4s 2024-09-16T12:22:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 63050K .......... .......... .......... .......... .......... 37% 25.7M 4s 2024-09-16T12:22:55.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 63100K .......... .......... .......... .......... .......... 37% 47.4M 4s 2024-09-16T12:22:55.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 63150K .......... .......... .......... .......... .......... 37% 19.7M 4s 2024-09-16T12:22:55.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 63200K .......... .......... .......... .......... .......... 37% 26.0M 4s 2024-09-16T12:22:55.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 63250K .......... .......... .......... .......... .......... 37% 24.2M 4s 2024-09-16T12:22:55.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 63300K .......... .......... .......... .......... .......... 37% 35.9M 4s 2024-09-16T12:22:55.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 63350K .......... .......... .......... .......... .......... 37% 42.8M 4s 2024-09-16T12:22:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 63400K .......... .......... .......... .......... .......... 37% 27.1M 4s 2024-09-16T12:22:55.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 63450K .......... .......... .......... .......... .......... 37% 32.7M 4s 2024-09-16T12:22:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 63500K .......... .......... .......... .......... .......... 37% 23.0M 4s 2024-09-16T12:22:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 63550K .......... .......... .......... .......... .......... 37% 28.0M 4s 2024-09-16T12:22:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 63600K .......... .......... .......... .......... .......... 37% 25.0M 4s 2024-09-16T12:22:55.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 63650K .......... .......... .......... .......... .......... 37% 38.5M 4s 2024-09-16T12:22:55.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 63700K .......... .......... .......... .......... .......... 37% 25.2M 4s 2024-09-16T12:22:55.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 63750K .......... .......... .......... .......... .......... 37% 45.5M 4s 2024-09-16T12:22:55.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 63800K .......... .......... .......... .......... .......... 37% 20.7M 4s 2024-09-16T12:22:55.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 63850K .......... .......... .......... .......... .......... 37% 51.2M 4s 2024-09-16T12:22:55.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 63900K .......... .......... .......... .......... .......... 37% 24.1M 4s 2024-09-16T12:22:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 63950K .......... .......... .......... .......... .......... 37% 25.3M 4s 2024-09-16T12:22:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 64000K .......... .......... .......... .......... .......... 37% 34.1M 4s 2024-09-16T12:22:55.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 64050K .......... .......... .......... .......... .......... 38% 23.1M 4s 2024-09-16T12:22:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 64100K .......... .......... .......... .......... .......... 38% 25.7M 4s 2024-09-16T12:22:55.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 64150K .......... .......... .......... .......... .......... 38% 52.9M 4s 2024-09-16T12:22:55.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 64200K .......... .......... .......... .......... .......... 38% 23.3M 4s 2024-09-16T12:22:55.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 64250K .......... .......... .......... .......... .......... 38% 43.5M 4s 2024-09-16T12:22:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 64300K .......... .......... .......... .......... .......... 38% 22.0M 4s 2024-09-16T12:22:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 64350K .......... .......... .......... .......... .......... 38% 26.7M 4s 2024-09-16T12:22:55.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 64400K .......... .......... .......... .......... .......... 38% 41.2M 4s 2024-09-16T12:22:55.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 64450K .......... .......... .......... .......... .......... 38% 24.7M 4s 2024-09-16T12:22:55.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 64500K .......... .......... .......... .......... .......... 38% 36.8M 4s 2024-09-16T12:22:55.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 64550K .......... .......... .......... .......... .......... 38% 24.8M 4s 2024-09-16T12:22:55.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 64600K .......... .......... .......... .......... .......... 38% 23.1M 4s 2024-09-16T12:22:55.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 64650K .......... .......... .......... .......... .......... 38% 52.7M 4s 2024-09-16T12:22:55.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 64700K .......... .......... .......... .......... .......... 38% 23.5M 4s 2024-09-16T12:22:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 64750K .......... .......... .......... .......... .......... 38% 24.3M 4s 2024-09-16T12:22:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 64800K .......... .......... .......... .......... .......... 38% 40.6M 4s 2024-09-16T12:22:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 64850K .......... .......... .......... .......... .......... 38% 25.6M 4s 2024-09-16T12:22:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 64900K .......... .......... .......... .......... .......... 38% 40.6M 4s 2024-09-16T12:22:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 64950K .......... .......... .......... .......... .......... 38% 24.5M 4s 2024-09-16T12:22:55.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 65000K .......... .......... .......... .......... .......... 38% 38.4M 4s 2024-09-16T12:22:55.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 65050K .......... .......... .......... .......... .......... 38% 24.6M 4s 2024-09-16T12:22:55.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 65100K .......... .......... .......... .......... .......... 38% 42.0M 4s 2024-09-16T12:22:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 65150K .......... .......... .......... .......... .......... 38% 10.8M 4s 2024-09-16T12:22:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 65200K .......... .......... .......... .......... .......... 38% 122M 4s 2024-09-16T12:22:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 65250K .......... .......... .......... .......... .......... 38% 66.2M 4s 2024-09-16T12:22:55.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 65300K .......... .......... .......... .......... .......... 38% 29.0M 4s 2024-09-16T12:22:55.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 65350K .......... .......... .......... .......... .......... 38% 24.9M 4s 2024-09-16T12:22:55.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 65400K .......... .......... .......... .......... .......... 38% 54.3M 4s 2024-09-16T12:22:55.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 65450K .......... .......... .......... .......... .......... 38% 21.3M 4s 2024-09-16T12:22:55.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 65500K .......... .......... .......... .......... .......... 38% 40.8M 4s 2024-09-16T12:22:55.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 65550K .......... .......... .......... .......... .......... 38% 23.4M 4s 2024-09-16T12:22:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 65600K .......... .......... .......... .......... .......... 38% 24.9M 4s 2024-09-16T12:22:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 65650K .......... .......... .......... .......... .......... 38% 47.3M 4s 2024-09-16T12:22:55.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 65700K .......... .......... .......... .......... .......... 39% 24.4M 4s 2024-09-16T12:22:55.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 65750K .......... .......... .......... .......... .......... 39% 33.7M 4s 2024-09-16T12:22:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 65800K .......... .......... .......... .......... .......... 39% 23.6M 4s 2024-09-16T12:22:55.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 65850K .......... .......... .......... .......... .......... 39% 25.5M 4s 2024-09-16T12:22:55.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 65900K .......... .......... .......... .......... .......... 39% 59.0M 4s 2024-09-16T12:22:55.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 65950K .......... .......... .......... .......... .......... 39% 22.4M 4s 2024-09-16T12:22:55.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 66000K .......... .......... .......... .......... .......... 39% 26.0M 4s 2024-09-16T12:22:55.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 66050K .......... .......... .......... .......... .......... 39% 33.0M 4s 2024-09-16T12:22:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 66100K .......... .......... .......... .......... .......... 39% 25.3M 4s 2024-09-16T12:22:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 66150K .......... .......... .......... .......... .......... 39% 43.0M 4s 2024-09-16T12:22:55.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 66200K .......... .......... .......... .......... .......... 39% 19.8M 4s 2024-09-16T12:22:55.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 66250K .......... .......... .......... .......... .......... 39% 39.6M 4s 2024-09-16T12:22:56.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 66300K .......... .......... .......... .......... .......... 39% 22.5M 4s 2024-09-16T12:22:56.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 66350K .......... .......... .......... .......... .......... 39% 24.4M 4s 2024-09-16T12:22:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 66400K .......... .......... .......... .......... .......... 39% 25.6M 4s 2024-09-16T12:22:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 66450K .......... .......... .......... .......... .......... 39% 64.4M 4s 2024-09-16T12:22:56.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 66500K .......... .......... .......... .......... .......... 39% 26.1M 4s 2024-09-16T12:22:56.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 66550K .......... .......... .......... .......... .......... 39% 26.6M 4s 2024-09-16T12:22:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 66600K .......... .......... .......... .......... .......... 39% 21.1M 4s 2024-09-16T12:22:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 66650K .......... .......... .......... .......... .......... 39% 47.5M 4s 2024-09-16T12:22:56.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 66700K .......... .......... .......... .......... .......... 39% 27.1M 4s 2024-09-16T12:22:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 66750K .......... .......... .......... .......... .......... 39% 20.1M 4s 2024-09-16T12:22:56.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 66800K .......... .......... .......... .......... .......... 39% 40.7M 4s 2024-09-16T12:22:56.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 66850K .......... .......... .......... .......... .......... 39% 24.2M 4s 2024-09-16T12:22:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 66900K .......... .......... .......... .......... .......... 39% 25.4M 4s 2024-09-16T12:22:56.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 66950K .......... .......... .......... .......... .......... 39% 49.4M 4s 2024-09-16T12:22:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 67000K .......... .......... .......... .......... .......... 39% 29.3M 4s 2024-09-16T12:22:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 67050K .......... .......... .......... .......... .......... 39% 26.4M 4s 2024-09-16T12:22:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 67100K .......... .......... .......... .......... .......... 39% 24.7M 4s 2024-09-16T12:22:56.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 67150K .......... .......... .......... .......... .......... 39% 25.5M 4s 2024-09-16T12:22:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 67200K .......... .......... .......... .......... .......... 39% 43.1M 4s 2024-09-16T12:22:56.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 67250K .......... .......... .......... .......... .......... 39% 25.9M 4s 2024-09-16T12:22:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 67300K .......... .......... .......... .......... .......... 39% 35.1M 4s 2024-09-16T12:22:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 67350K .......... .......... .......... .......... .......... 39% 23.8M 4s 2024-09-16T12:22:56.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 67400K .......... .......... .......... .......... .......... 40% 53.3M 4s 2024-09-16T12:22:56.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 67450K .......... .......... .......... .......... .......... 40% 24.2M 4s 2024-09-16T12:22:56.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 67500K .......... .......... .......... .......... .......... 40% 31.9M 4s 2024-09-16T12:22:56.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 67550K .......... .......... .......... .......... .......... 40% 19.2M 4s 2024-09-16T12:22:56.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 67600K .......... .......... .......... .......... .......... 40% 34.4M 4s 2024-09-16T12:22:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 67650K .......... .......... .......... .......... .......... 40% 26.0M 4s 2024-09-16T12:22:56.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 67700K .......... .......... .......... .......... .......... 40% 58.5M 4s 2024-09-16T12:22:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 67750K .......... .......... .......... .......... .......... 40% 24.4M 4s 2024-09-16T12:22:56.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 67800K .......... .......... .......... .......... .......... 40% 30.4M 4s 2024-09-16T12:22:56.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 67850K .......... .......... .......... .......... .......... 40% 24.1M 4s 2024-09-16T12:22:56.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 67900K .......... .......... .......... .......... .......... 40% 46.7M 4s 2024-09-16T12:22:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 67950K .......... .......... .......... .......... .......... 40% 23.4M 4s 2024-09-16T12:22:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 68000K .......... .......... .......... .......... .......... 40% 26.8M 4s 2024-09-16T12:22:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 68050K .......... .......... .......... .......... .......... 40% 36.3M 4s 2024-09-16T12:22:56.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 68100K .......... .......... .......... .......... .......... 40% 24.1M 4s 2024-09-16T12:22:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 68150K .......... .......... .......... .......... .......... 40% 24.9M 4s 2024-09-16T12:22:56.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 68200K .......... .......... .......... .......... .......... 40% 54.6M 4s 2024-09-16T12:22:56.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 68250K .......... .......... .......... .......... .......... 40% 23.4M 4s 2024-09-16T12:22:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 68300K .......... .......... .......... .......... .......... 40% 41.4M 3s 2024-09-16T12:22:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 68350K .......... .......... .......... .......... .......... 40% 23.1M 3s 2024-09-16T12:22:56.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 68400K .......... .......... .......... .......... .......... 40% 23.7M 3s 2024-09-16T12:22:56.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 68450K .......... .......... .......... .......... .......... 40% 48.7M 3s 2024-09-16T12:22:56.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 68500K .......... .......... .......... .......... .......... 40% 24.8M 3s 2024-09-16T12:22:56.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 68550K .......... .......... .......... .......... .......... 40% 36.7M 3s 2024-09-16T12:22:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 68600K .......... .......... .......... .......... .......... 40% 23.3M 3s 2024-09-16T12:22:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 68650K .......... .......... .......... .......... .......... 40% 24.0M 3s 2024-09-16T12:22:56.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 68700K .......... .......... .......... .......... .......... 40% 45.4M 3s 2024-09-16T12:22:56.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 68750K .......... .......... .......... .......... .......... 40% 24.3M 3s 2024-09-16T12:22:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 68800K .......... .......... .......... .......... .......... 40% 25.2M 3s 2024-09-16T12:22:56.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 68850K .......... .......... .......... .......... .......... 40% 35.6M 3s 2024-09-16T12:22:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 68900K .......... .......... .......... .......... .......... 40% 26.6M 3s 2024-09-16T12:22:56.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 68950K .......... .......... .......... .......... .......... 40% 41.1M 3s 2024-09-16T12:22:56.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 69000K .......... .......... .......... .......... .......... 40% 23.8M 3s 2024-09-16T12:22:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 69050K .......... .......... .......... .......... .......... 40% 41.7M 3s 2024-09-16T12:22:56.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 69100K .......... .......... .......... .......... .......... 41% 24.9M 3s 2024-09-16T12:22:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 69150K .......... .......... .......... .......... .......... 41% 23.4M 3s 2024-09-16T12:22:56.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 69200K .......... .......... .......... .......... .......... 41% 23.9M 3s 2024-09-16T12:22:56.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 69250K .......... .......... .......... .......... .......... 41% 45.2M 3s 2024-09-16T12:22:56.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 69300K .......... .......... .......... .......... .......... 41% 36.4M 3s 2024-09-16T12:22:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 69350K .......... .......... .......... .......... .......... 41% 30.9M 3s 2024-09-16T12:22:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 69400K .......... .......... .......... .......... .......... 41% 18.8M 3s 2024-09-16T12:22:56.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 69450K .......... .......... .......... .......... .......... 41% 78.7M 3s 2024-09-16T12:22:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 69500K .......... .......... .......... .......... .......... 41% 24.1M 3s 2024-09-16T12:22:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 69550K .......... .......... .......... .......... .......... 41% 24.8M 3s 2024-09-16T12:22:56.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 69600K .......... .......... .......... .......... .......... 41% 26.5M 3s 2024-09-16T12:22:56.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 69650K .......... .......... .......... .......... .......... 41% 29.3M 3s 2024-09-16T12:22:56.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 69700K .......... .......... .......... .......... .......... 41% 42.6M 3s 2024-09-16T12:22:56.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 69750K .......... .......... .......... .......... .......... 41% 25.7M 3s 2024-09-16T12:22:56.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 69800K .......... .......... .......... .......... .......... 41% 24.6M 3s 2024-09-16T12:22:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 69850K .......... .......... .......... .......... .......... 41% 25.3M 3s 2024-09-16T12:22:56.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 69900K .......... .......... .......... .......... .......... 41% 40.9M 3s 2024-09-16T12:22:56.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 69950K .......... .......... .......... .......... .......... 41% 24.7M 3s 2024-09-16T12:22:56.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 70000K .......... .......... .......... .......... .......... 41% 24.6M 3s 2024-09-16T12:22:56.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 70050K .......... .......... .......... .......... .......... 41% 45.2M 3s 2024-09-16T12:22:56.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 70100K .......... .......... .......... .......... .......... 41% 27.8M 3s 2024-09-16T12:22:56.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 70150K .......... .......... .......... .......... .......... 41% 25.2M 3s 2024-09-16T12:22:56.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 70200K .......... .......... .......... .......... .......... 41% 42.2M 3s 2024-09-16T12:22:56.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 70250K .......... .......... .......... .......... .......... 41% 25.1M 3s 2024-09-16T12:22:56.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 70300K .......... .......... .......... .......... .......... 41% 33.4M 3s 2024-09-16T12:22:56.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 70350K .......... .......... .......... .......... .......... 41% 23.7M 3s 2024-09-16T12:22:56.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 70400K .......... .......... .......... .......... .......... 41% 25.0M 3s 2024-09-16T12:22:56.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 70450K .......... .......... .......... .......... .......... 41% 28.1M 3s 2024-09-16T12:22:56.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 70500K .......... .......... .......... .......... .......... 41% 39.9M 3s 2024-09-16T12:22:56.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 70550K .......... .......... .......... .......... .......... 41% 23.3M 3s 2024-09-16T12:22:56.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 70600K .......... .......... .......... .......... .......... 41% 37.6M 3s 2024-09-16T12:22:56.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 70650K .......... .......... .......... .......... .......... 41% 25.2M 3s 2024-09-16T12:22:56.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 70700K .......... .......... .......... .......... .......... 41% 50.5M 3s 2024-09-16T12:22:56.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 70750K .......... .......... .......... .......... .......... 42% 26.0M 3s 2024-09-16T12:22:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 70800K .......... .......... .......... .......... .......... 42% 22.7M 3s 2024-09-16T12:22:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 70850K .......... .......... .......... .......... .......... 42% 34.3M 3s 2024-09-16T12:22:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 70900K .......... .......... .......... .......... .......... 42% 24.1M 3s 2024-09-16T12:22:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 70950K .......... .......... .......... .......... .......... 42% 24.2M 3s 2024-09-16T12:22:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 71000K .......... .......... .......... .......... .......... 42% 57.6M 3s 2024-09-16T12:22:56.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 71050K .......... .......... .......... .......... .......... 42% 22.4M 3s 2024-09-16T12:22:56.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 71100K .......... .......... .......... .......... .......... 42% 42.2M 3s 2024-09-16T12:22:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 71150K .......... .......... .......... .......... .......... 42% 22.5M 3s 2024-09-16T12:22:56.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 71200K .......... .......... .......... .......... .......... 42% 24.4M 3s 2024-09-16T12:22:56.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 71250K .......... .......... .......... .......... .......... 42% 49.3M 3s 2024-09-16T12:22:56.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 71300K .......... .......... .......... .......... .......... 42% 25.3M 3s 2024-09-16T12:22:56.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 71350K .......... .......... .......... .......... .......... 42% 38.9M 3s 2024-09-16T12:22:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 71400K .......... .......... .......... .......... .......... 42% 27.3M 3s 2024-09-16T12:22:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 71450K .......... .......... .......... .......... .......... 42% 20.5M 3s 2024-09-16T12:22:56.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 71500K .......... .......... .......... .......... .......... 42% 51.8M 3s 2024-09-16T12:22:56.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 71550K .......... .......... .......... .......... .......... 42% 21.7M 3s 2024-09-16T12:22:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 71600K .......... .......... .......... .......... .......... 42% 24.9M 3s 2024-09-16T12:22:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 71650K .......... .......... .......... .......... .......... 42% 36.1M 3s 2024-09-16T12:22:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 71700K .......... .......... .......... .......... .......... 42% 24.0M 3s 2024-09-16T12:22:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 71750K .......... .......... .......... .......... .......... 42% 44.1M 3s 2024-09-16T12:22:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 71800K .......... .......... .......... .......... .......... 42% 23.4M 3s 2024-09-16T12:22:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 71850K .......... .......... .......... .......... .......... 42% 39.3M 3s 2024-09-16T12:22:56.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 71900K .......... .......... .......... .......... .......... 42% 26.5M 3s 2024-09-16T12:22:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 71950K .......... .......... .......... .......... .......... 42% 21.8M 3s 2024-09-16T12:22:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 72000K .......... .......... .......... .......... .......... 42% 31.6M 3s 2024-09-16T12:22:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 72050K .......... .......... .......... .......... .......... 42% 37.3M 3s 2024-09-16T12:22:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 72100K .......... .......... .......... .......... .......... 42% 23.5M 3s 2024-09-16T12:22:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 72150K .......... .......... .......... .......... .......... 42% 36.0M 3s 2024-09-16T12:22:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 72200K .......... .......... .......... .......... .......... 42% 25.1M 3s 2024-09-16T12:22:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 72250K .......... .......... .......... .......... .......... 42% 46.7M 3s 2024-09-16T12:22:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 72300K .......... .......... .......... .......... .......... 42% 23.8M 3s 2024-09-16T12:22:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 72350K .......... .......... .......... .......... .......... 42% 23.7M 3s 2024-09-16T12:22:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 72400K .......... .......... .......... .......... .......... 42% 36.5M 3s 2024-09-16T12:22:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 72450K .......... .......... .......... .......... .......... 43% 25.9M 3s 2024-09-16T12:22:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 72500K .......... .......... .......... .......... .......... 43% 22.9M 3s 2024-09-16T12:22:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 72550K .......... .......... .......... .......... .......... 43% 51.6M 3s 2024-09-16T12:22:56.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 72600K .......... .......... .......... .......... .......... 43% 36.2M 3s 2024-09-16T12:22:56.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 72650K .......... .......... .......... .......... .......... 43% 24.0M 3s 2024-09-16T12:22:56.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 72700K .......... .......... .......... .......... .......... 43% 24.5M 3s 2024-09-16T12:22:56.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 72750K .......... .......... .......... .......... .......... 43% 27.7M 3s 2024-09-16T12:22:56.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 72800K .......... .......... .......... .......... .......... 43% 40.1M 3s 2024-09-16T12:22:56.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 72850K .......... .......... .......... .......... .......... 43% 23.7M 3s 2024-09-16T12:22:56.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 72900K .......... .......... .......... .......... .......... 43% 37.0M 3s 2024-09-16T12:22:56.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 72950K .......... .......... .......... .......... .......... 43% 24.7M 3s 2024-09-16T12:22:56.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 73000K .......... .......... .......... .......... .......... 43% 47.6M 3s 2024-09-16T12:22:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 73050K .......... .......... .......... .......... .......... 43% 23.8M 3s 2024-09-16T12:22:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 73100K .......... .......... .......... .......... .......... 43% 38.8M 3s 2024-09-16T12:22:56.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 73150K .......... .......... .......... .......... .......... 43% 22.5M 3s 2024-09-16T12:22:56.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 73200K .......... .......... .......... .......... .......... 43% 23.9M 3s 2024-09-16T12:22:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 73250K .......... .......... .......... .......... .......... 43% 25.9M 3s 2024-09-16T12:22:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 73300K .......... .......... .......... .......... .......... 43% 50.7M 3s 2024-09-16T12:22:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 73350K .......... .......... .......... .......... .......... 43% 23.3M 3s 2024-09-16T12:22:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 73400K .......... .......... .......... .......... .......... 43% 39.3M 3s 2024-09-16T12:22:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 73450K .......... .......... .......... .......... .......... 43% 24.3M 3s 2024-09-16T12:22:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 73500K .......... .......... .......... .......... .......... 43% 44.2M 3s 2024-09-16T12:22:56.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 73550K .......... .......... .......... .......... .......... 43% 23.8M 3s 2024-09-16T12:22:56.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 73600K .......... .......... .......... .......... .......... 43% 25.7M 3s 2024-09-16T12:22:56.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 73650K .......... .......... .......... .......... .......... 43% 36.0M 3s 2024-09-16T12:22:56.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 73700K .......... .......... .......... .......... .......... 43% 23.5M 3s 2024-09-16T12:22:56.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 73750K .......... .......... .......... .......... .......... 43% 23.9M 3s 2024-09-16T12:22:56.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 73800K .......... .......... .......... .......... .......... 43% 53.5M 3s 2024-09-16T12:22:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 73850K .......... .......... .......... .......... .......... 43% 23.1M 3s 2024-09-16T12:22:56.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 73900K .......... .......... .......... .......... .......... 43% 38.1M 3s 2024-09-16T12:22:56.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 73950K .......... .......... .......... .......... .......... 43% 23.5M 3s 2024-09-16T12:22:56.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 74000K .......... .......... .......... .......... .......... 43% 24.5M 3s 2024-09-16T12:22:56.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 74050K .......... .......... .......... .......... .......... 43% 49.9M 3s 2024-09-16T12:22:56.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 74100K .......... .......... .......... .......... .......... 43% 24.3M 3s 2024-09-16T12:22:56.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 74150K .......... .......... .......... .......... .......... 44% 34.9M 3s 2024-09-16T12:22:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 74200K .......... .......... .......... .......... .......... 44% 25.4M 3s 2024-09-16T12:22:56.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 74250K .......... .......... .......... .......... .......... 44% 23.0M 3s 2024-09-16T12:22:56.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 74300K .......... .......... .......... .......... .......... 44% 57.4M 3s 2024-09-16T12:22:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 74350K .......... .......... .......... .......... .......... 44% 24.1M 3s 2024-09-16T12:22:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 74400K .......... .......... .......... .......... .......... 44% 23.7M 3s 2024-09-16T12:22:56.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 74450K .......... .......... .......... .......... .......... 44% 35.5M 3s 2024-09-16T12:22:56.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 74500K .......... .......... .......... .......... .......... 44% 25.0M 3s 2024-09-16T12:22:56.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 74550K .......... .......... .......... .......... .......... 44% 45.1M 3s 2024-09-16T12:22:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 74600K .......... .......... .......... .......... .......... 44% 25.1M 3s 2024-09-16T12:22:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 74650K .......... .......... .......... .......... .......... 44% 34.9M 3s 2024-09-16T12:22:56.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 74700K .......... .......... .......... .......... .......... 44% 25.4M 3s 2024-09-16T12:22:56.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 74750K .......... .......... .......... .......... .......... 44% 23.8M 3s 2024-09-16T12:22:56.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 74800K .......... .......... .......... .......... .......... 44% 22.9M 3s 2024-09-16T12:22:56.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 74850K .......... .......... .......... .......... .......... 44% 56.1M 3s 2024-09-16T12:22:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 74900K .......... .......... .......... .......... .......... 44% 35.1M 3s 2024-09-16T12:22:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 74950K .......... .......... .......... .......... .......... 44% 22.7M 3s 2024-09-16T12:22:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 75000K .......... .......... .......... .......... .......... 44% 23.9M 3s 2024-09-16T12:22:56.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 75050K .......... .......... .......... .......... .......... 44% 58.9M 3s 2024-09-16T12:22:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 75100K .......... .......... .......... .......... .......... 44% 24.7M 3s 2024-09-16T12:22:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 75150K .......... .......... .......... .......... .......... 44% 23.9M 3s 2024-09-16T12:22:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 75200K .......... .......... .......... .......... .......... 44% 36.9M 3s 2024-09-16T12:22:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 75250K .......... .......... .......... .......... .......... 44% 23.6M 3s 2024-09-16T12:22:56.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 75300K .......... .......... .......... .......... .......... 44% 52.6M 3s 2024-09-16T12:22:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 75350K .......... .......... .......... .......... .......... 44% 24.4M 3s 2024-09-16T12:22:56.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 75400K .......... .......... .......... .......... .......... 44% 34.0M 3s 2024-09-16T12:22:56.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 75450K .......... .......... .......... .......... .......... 44% 24.9M 3s 2024-09-16T12:22:56.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 75500K .......... .......... .......... .......... .......... 44% 24.6M 3s 2024-09-16T12:22:56.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 75550K .......... .......... .......... .......... .......... 44% 23.9M 3s 2024-09-16T12:22:56.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 75600K .......... .......... .......... .......... .......... 44% 44.5M 3s 2024-09-16T12:22:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 75650K .......... .......... .......... .......... .......... 44% 26.1M 3s 2024-09-16T12:22:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 75700K .......... .......... .......... .......... .......... 44% 35.2M 3s 2024-09-16T12:22:56.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 75750K .......... .......... .......... .......... .......... 44% 24.4M 3s 2024-09-16T12:22:56.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 75800K .......... .......... .......... .......... .......... 44% 48.3M 3s 2024-09-16T12:22:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 75850K .......... .......... .......... .......... .......... 45% 24.4M 3s 2024-09-16T12:22:56.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 75900K .......... .......... .......... .......... .......... 45% 36.7M 3s 2024-09-16T12:22:56.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 75950K .......... .......... .......... .......... .......... 45% 23.0M 3s 2024-09-16T12:22:56.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 76000K .......... .......... .......... .......... .......... 45% 24.2M 3s 2024-09-16T12:22:56.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 76050K .......... .......... .......... .......... .......... 45% 25.6M 3s 2024-09-16T12:22:56.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 76100K .......... .......... .......... .......... .......... 45% 50.9M 3s 2024-09-16T12:22:56.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 76150K .......... .......... .......... .......... .......... 45% 23.6M 3s 2024-09-16T12:22:56.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 76200K .......... .......... .......... .......... .......... 45% 37.0M 3s 2024-09-16T12:22:56.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 76250K .......... .......... .......... .......... .......... 45% 24.9M 3s 2024-09-16T12:22:56.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 76300K .......... .......... .......... .......... .......... 45% 46.3M 3s 2024-09-16T12:22:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 76350K .......... .......... .......... .......... .......... 45% 26.9M 3s 2024-09-16T12:22:56.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 76400K .......... .......... .......... .......... .......... 45% 22.2M 3s 2024-09-16T12:22:56.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 76450K .......... .......... .......... .......... .......... 45% 37.0M 3s 2024-09-16T12:22:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 76500K .......... .......... .......... .......... .......... 45% 23.1M 3s 2024-09-16T12:22:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 76550K .......... .......... .......... .......... .......... 45% 25.1M 3s 2024-09-16T12:22:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 76600K .......... .......... .......... .......... .......... 45% 52.0M 3s 2024-09-16T12:22:56.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 76650K .......... .......... .......... .......... .......... 45% 23.5M 3s 2024-09-16T12:22:56.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 76700K .......... .......... .......... .......... .......... 45% 35.9M 3s 2024-09-16T12:22:56.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 76750K .......... .......... .......... .......... .......... 45% 24.2M 3s 2024-09-16T12:22:56.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 76800K .......... .......... .......... .......... .......... 45% 25.2M 3s 2024-09-16T12:22:56.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 76850K .......... .......... .......... .......... .......... 45% 47.3M 3s 2024-09-16T12:22:56.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 76900K .......... .......... .......... .......... .......... 45% 19.7M 3s 2024-09-16T12:22:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 76950K .......... .......... .......... .......... .......... 45% 36.2M 3s 2024-09-16T12:22:56.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 77000K .......... .......... .......... .......... .......... 45% 24.9M 3s 2024-09-16T12:22:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 77050K .......... .......... .......... .......... .......... 45% 22.8M 3s 2024-09-16T12:22:56.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 77100K .......... .......... .......... .......... .......... 45% 50.7M 3s 2024-09-16T12:22:56.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 77150K .......... .......... .......... .......... .......... 45% 24.9M 3s 2024-09-16T12:22:56.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 77200K .......... .......... .......... .......... .......... 45% 22.5M 3s 2024-09-16T12:22:56.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 77250K .......... .......... .......... .......... .......... 45% 37.8M 3s 2024-09-16T12:22:56.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 77300K .......... .......... .......... .......... .......... 45% 20.3M 3s 2024-09-16T12:22:56.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 77350K .......... .......... .......... .......... .......... 45% 43.0M 3s 2024-09-16T12:22:56.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 77400K .......... .......... .......... .......... .......... 45% 23.8M 3s 2024-09-16T12:22:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 77450K .......... .......... .......... .......... .......... 45% 36.9M 3s 2024-09-16T12:22:56.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 77500K .......... .......... .......... .......... .......... 46% 29.9M 3s 2024-09-16T12:22:56.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 77550K .......... .......... .......... .......... .......... 46% 20.5M 3s 2024-09-16T12:22:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 77600K .......... .......... .......... .......... .......... 46% 26.3M 3s 2024-09-16T12:22:56.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 77650K .......... .......... .......... .......... .......... 46% 37.5M 3s 2024-09-16T12:22:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 77700K .......... .......... .......... .......... .......... 46% 38.7M 3s 2024-09-16T12:22:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 77750K .......... .......... .......... .......... .......... 46% 27.3M 3s 2024-09-16T12:22:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 77800K .......... .......... .......... .......... .......... 46% 24.5M 3s 2024-09-16T12:22:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 77850K .......... .......... .......... .......... .......... 46% 38.7M 3s 2024-09-16T12:22:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 77900K .......... .......... .......... .......... .......... 46% 27.8M 3s 2024-09-16T12:22:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 77950K .......... .......... .......... .......... .......... 46% 26.6M 3s 2024-09-16T12:22:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 78000K .......... .......... .......... .......... .......... 46% 27.9M 3s 2024-09-16T12:22:56.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 78050K .......... .......... .......... .......... .......... 46% 30.5M 3s 2024-09-16T12:22:56.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 78100K .......... .......... .......... .......... .......... 46% 34.0M 3s 2024-09-16T12:22:56.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 78150K .......... .......... .......... .......... .......... 46% 25.0M 3s 2024-09-16T12:22:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 78200K .......... .......... .......... .......... .......... 46% 32.9M 3s 2024-09-16T12:22:56.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 78250K .......... .......... .......... .......... .......... 46% 25.6M 3s 2024-09-16T12:22:56.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 78300K .......... .......... .......... .......... .......... 46% 25.5M 3s 2024-09-16T12:22:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 78350K .......... .......... .......... .......... .......... 46% 23.9M 3s 2024-09-16T12:22:56.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 78400K .......... .......... .......... .......... .......... 46% 42.9M 3s 2024-09-16T12:22:56.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 78450K .......... .......... .......... .......... .......... 46% 26.1M 3s 2024-09-16T12:22:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 78500K .......... .......... .......... .......... .......... 46% 36.9M 3s 2024-09-16T12:22:56.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 78550K .......... .......... .......... .......... .......... 46% 23.7M 3s 2024-09-16T12:22:56.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 78600K .......... .......... .......... .......... .......... 46% 42.0M 3s 2024-09-16T12:22:56.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 78650K .......... .......... .......... .......... .......... 46% 25.0M 3s 2024-09-16T12:22:56.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 78700K .......... .......... .......... .......... .......... 46% 24.1M 3s 2024-09-16T12:22:56.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 78750K .......... .......... .......... .......... .......... 46% 36.8M 3s 2024-09-16T12:22:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 78800K .......... .......... .......... .......... .......... 46% 23.1M 3s 2024-09-16T12:22:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 78850K .......... .......... .......... .......... .......... 46% 26.2M 3s 2024-09-16T12:22:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 78900K .......... .......... .......... .......... .......... 46% 45.2M 3s 2024-09-16T12:22:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 78950K .......... .......... .......... .......... .......... 46% 24.5M 3s 2024-09-16T12:22:56.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 79000K .......... .......... .......... .......... .......... 46% 36.7M 3s 2024-09-16T12:22:56.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 79050K .......... .......... .......... .......... .......... 46% 25.1M 3s 2024-09-16T12:22:56.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 79100K .......... .......... .......... .......... .......... 46% 46.1M 3s 2024-09-16T12:22:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 79150K .......... .......... .......... .......... .......... 46% 24.3M 3s 2024-09-16T12:22:56.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 79200K .......... .......... .......... .......... .......... 47% 25.0M 3s 2024-09-16T12:22:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 79250K .......... .......... .......... .......... .......... 47% 28.5M 3s 2024-09-16T12:22:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 79300K .......... .......... .......... .......... .......... 47% 32.5M 3s 2024-09-16T12:22:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 79350K .......... .......... .......... .......... .......... 47% 24.0M 3s 2024-09-16T12:22:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 79400K .......... .......... .......... .......... .......... 47% 45.1M 3s 2024-09-16T12:22:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 79450K .......... .......... .......... .......... .......... 47% 22.7M 3s 2024-09-16T12:22:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 79500K .......... .......... .......... .......... .......... 47% 40.6M 3s 2024-09-16T12:22:56.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 79550K .......... .......... .......... .......... .......... 47% 24.0M 3s 2024-09-16T12:22:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 79600K .......... .......... .......... .......... .......... 47% 24.9M 3s 2024-09-16T12:22:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 79650K .......... .......... .......... .......... .......... 47% 42.7M 3s 2024-09-16T12:22:56.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 79700K .......... .......... .......... .......... .......... 47% 25.8M 3s 2024-09-16T12:22:56.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 79750K .......... .......... .......... .......... .......... 47% 35.8M 3s 2024-09-16T12:22:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 79800K .......... .......... .......... .......... .......... 47% 25.4M 3s 2024-09-16T12:22:56.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 79850K .......... .......... .......... .......... .......... 47% 45.4M 3s 2024-09-16T12:22:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 79900K .......... .......... .......... .......... .......... 47% 24.5M 3s 2024-09-16T12:22:56.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 79950K .......... .......... .......... .......... .......... 47% 23.3M 3s 2024-09-16T12:22:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 80000K .......... .......... .......... .......... .......... 47% 27.2M 3s 2024-09-16T12:22:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 80050K .......... .......... .......... .......... .......... 47% 31.8M 3s 2024-09-16T12:22:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 80100K .......... .......... .......... .......... .......... 47% 25.9M 3s 2024-09-16T12:22:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 80150K .......... .......... .......... .......... .......... 47% 39.3M 3s 2024-09-16T12:22:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 80200K .......... .......... .......... .......... .......... 47% 25.2M 3s 2024-09-16T12:22:56.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 80250K .......... .......... .......... .......... .......... 47% 37.3M 3s 2024-09-16T12:22:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 80300K .......... .......... .......... .......... .......... 47% 24.7M 3s 2024-09-16T12:22:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 80350K .......... .......... .......... .......... .......... 47% 24.5M 3s 2024-09-16T12:22:56.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 80400K .......... .......... .......... .......... .......... 47% 44.5M 3s 2024-09-16T12:22:56.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 80450K .......... .......... .......... .......... .......... 47% 21.0M 3s 2024-09-16T12:22:56.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 80500K .......... .......... .......... .......... .......... 47% 34.4M 3s 2024-09-16T12:22:56.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 80550K .......... .......... .......... .......... .......... 47% 23.8M 3s 2024-09-16T12:22:56.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 80600K .......... .......... .......... .......... .......... 47% 24.8M 3s 2024-09-16T12:22:56.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 80650K .......... .......... .......... .......... .......... 47% 44.0M 3s 2024-09-16T12:22:56.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 80700K .......... .......... .......... .......... .......... 47% 27.4M 3s 2024-09-16T12:22:56.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 80750K .......... .......... .......... .......... .......... 47% 22.3M 3s 2024-09-16T12:22:56.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 80800K .......... .......... .......... .......... .......... 47% 38.1M 3s 2024-09-16T12:22:56.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 80850K .......... .......... .......... .......... .......... 47% 20.2M 3s 2024-09-16T12:22:56.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 80900K .......... .......... .......... .......... .......... 48% 44.4M 3s 2024-09-16T12:22:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 80950K .......... .......... .......... .......... .......... 48% 24.3M 3s 2024-09-16T12:22:56.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 81000K .......... .......... .......... .......... .......... 48% 37.5M 3s 2024-09-16T12:22:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 81050K .......... .......... .......... .......... .......... 48% 25.7M 3s 2024-09-16T12:22:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 81100K .......... .......... .......... .......... .......... 48% 22.7M 3s 2024-09-16T12:22:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 81150K .......... .......... .......... .......... .......... 48% 24.2M 3s 2024-09-16T12:22:56.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 81200K .......... .......... .......... .......... .......... 48% 50.8M 3s 2024-09-16T12:22:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 81250K .......... .......... .......... .......... .......... 48% 25.3M 3s 2024-09-16T12:22:56.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 81300K .......... .......... .......... .......... .......... 48% 36.2M 3s 2024-09-16T12:22:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 81350K .......... .......... .......... .......... .......... 48% 23.1M 3s 2024-09-16T12:22:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 81400K .......... .......... .......... .......... .......... 48% 49.6M 3s 2024-09-16T12:22:56.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 81450K .......... .......... .......... .......... .......... 48% 24.6M 3s 2024-09-16T12:22:56.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 81500K .......... .......... .......... .......... .......... 48% 33.8M 3s 2024-09-16T12:22:56.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 81550K .......... .......... .......... .......... .......... 48% 25.8M 3s 2024-09-16T12:22:56.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 81600K .......... .......... .......... .......... .......... 48% 22.5M 3s 2024-09-16T12:22:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 81650K .......... .......... .......... .......... .......... 48% 25.9M 3s 2024-09-16T12:22:56.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 81700K .......... .......... .......... .......... .......... 48% 55.0M 3s 2024-09-16T12:22:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 81750K .......... .......... .......... .......... .......... 48% 22.1M 3s 2024-09-16T12:22:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 81800K .......... .......... .......... .......... .......... 48% 37.5M 3s 2024-09-16T12:22:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 81850K .......... .......... .......... .......... .......... 48% 24.4M 3s 2024-09-16T12:22:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 81900K .......... .......... .......... .......... .......... 48% 46.7M 3s 2024-09-16T12:22:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 81950K .......... .......... .......... .......... .......... 48% 24.0M 3s 2024-09-16T12:22:56.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 82000K .......... .......... .......... .......... .......... 48% 25.0M 3s 2024-09-16T12:22:56.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 82050K .......... .......... .......... .......... .......... 48% 35.5M 3s 2024-09-16T12:22:56.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 82100K .......... .......... .......... .......... .......... 48% 26.3M 3s 2024-09-16T12:22:56.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 82150K .......... .......... .......... .......... .......... 48% 22.7M 3s 2024-09-16T12:22:56.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 82200K .......... .......... .......... .......... .......... 48% 48.1M 3s 2024-09-16T12:22:56.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 82250K .......... .......... .......... .......... .......... 48% 34.4M 3s 2024-09-16T12:22:56.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 82300K .......... .......... .......... .......... .......... 48% 24.1M 3s 2024-09-16T12:22:56.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 82350K .......... .......... .......... .......... .......... 48% 24.1M 3s 2024-09-16T12:22:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 82400K .......... .......... .......... .......... .......... 48% 24.2M 3s 2024-09-16T12:22:56.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 82450K .......... .......... .......... .......... .......... 48% 44.3M 3s 2024-09-16T12:22:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 82500K .......... .......... .......... .......... .......... 48% 26.0M 3s 2024-09-16T12:22:56.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 82550K .......... .......... .......... .......... .......... 49% 35.3M 3s 2024-09-16T12:22:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 82600K .......... .......... .......... .......... .......... 49% 25.4M 3s 2024-09-16T12:22:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 82650K .......... .......... .......... .......... .......... 49% 48.8M 3s 2024-09-16T12:22:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 82700K .......... .......... .......... .......... .......... 49% 25.1M 3s 2024-09-16T12:22:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 82750K .......... .......... .......... .......... .......... 49% 23.3M 3s 2024-09-16T12:22:56.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 82800K .......... .......... .......... .......... .......... 49% 22.7M 3s 2024-09-16T12:22:56.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 82850K .......... .......... .......... .......... .......... 49% 38.7M 3s 2024-09-16T12:22:56.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 82900K .......... .......... .......... .......... .......... 49% 25.4M 3s 2024-09-16T12:22:56.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 82950K .......... .......... .......... .......... .......... 49% 50.4M 3s 2024-09-16T12:22:56.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 83000K .......... .......... .......... .......... .......... 49% 22.3M 3s 2024-09-16T12:22:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 83050K .......... .......... .......... .......... .......... 49% 41.3M 3s 2024-09-16T12:22:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 83100K .......... .......... .......... .......... .......... 49% 24.5M 3s 2024-09-16T12:22:56.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 83150K .......... .......... .......... .......... .......... 49% 24.3M 3s 2024-09-16T12:22:56.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 83200K .......... .......... .......... .......... .......... 49% 44.4M 3s 2024-09-16T12:22:56.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 83250K .......... .......... .......... .......... .......... 49% 25.4M 3s 2024-09-16T12:22:56.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 83300K .......... .......... .......... .......... .......... 49% 33.7M 3s 2024-09-16T12:22:56.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 83350K .......... .......... .......... .......... .......... 49% 24.4M 3s 2024-09-16T12:22:56.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 83400K .......... .......... .......... .......... .......... 49% 24.5M 3s 2024-09-16T12:22:56.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 83450K .......... .......... .......... .......... .......... 49% 46.4M 3s 2024-09-16T12:22:56.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 83500K .......... .......... .......... .......... .......... 49% 24.8M 3s 2024-09-16T12:22:56.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 83550K .......... .......... .......... .......... .......... 49% 36.6M 3s 2024-09-16T12:22:56.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 83600K .......... .......... .......... .......... .......... 49% 23.4M 3s 2024-09-16T12:22:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 83650K .......... .......... .......... .......... .......... 49% 25.8M 3s 2024-09-16T12:22:56.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 83700K .......... .......... .......... .......... .......... 49% 44.1M 3s 2024-09-16T12:22:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 83750K .......... .......... .......... .......... .......... 49% 24.9M 3s 2024-09-16T12:22:56.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 83800K .......... .......... .......... .......... .......... 49% 37.3M 3s 2024-09-16T12:22:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 83850K .......... .......... .......... .......... .......... 49% 22.6M 3s 2024-09-16T12:22:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 83900K .......... .......... .......... .......... .......... 49% 25.8M 3s 2024-09-16T12:22:56.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 83950K .......... .......... .......... .......... .......... 49% 20.9M 3s 2024-09-16T12:22:56.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 84000K .......... .......... .......... .......... .......... 49% 45.1M 3s 2024-09-16T12:22:56.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 84050K .......... .......... .......... .......... .......... 49% 24.3M 3s 2024-09-16T12:22:56.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 84100K .......... .......... .......... .......... .......... 49% 37.0M 3s 2024-09-16T12:22:56.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 84150K .......... .......... .......... .......... .......... 49% 24.8M 3s 2024-09-16T12:22:56.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 84200K .......... .......... .......... .......... .......... 49% 47.2M 3s 2024-09-16T12:22:56.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 84250K .......... .......... .......... .......... .......... 50% 28.5M 3s 2024-09-16T12:22:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 84300K .......... .......... .......... .......... .......... 50% 29.9M 3s 2024-09-16T12:22:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 84350K .......... .......... .......... .......... .......... 50% 24.2M 3s 2024-09-16T12:22:56.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 84400K .......... .......... .......... .......... .......... 50% 20.5M 3s 2024-09-16T12:22:56.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 84450K .......... .......... .......... .......... .......... 50% 24.2M 3s 2024-09-16T12:22:56.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 84500K .......... .......... .......... .......... .......... 50% 50.4M 3s 2024-09-16T12:22:56.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 84550K .......... .......... .......... .......... .......... 50% 22.3M 3s 2024-09-16T12:22:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 84600K .......... .......... .......... .......... .......... 50% 41.9M 3s 2024-09-16T12:22:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 84650K .......... .......... .......... .......... .......... 50% 23.1M 3s 2024-09-16T12:22:56.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 84700K .......... .......... .......... .......... .......... 50% 48.2M 3s 2024-09-16T12:22:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 84750K .......... .......... .......... .......... .......... 50% 24.1M 3s 2024-09-16T12:22:56.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 84800K .......... .......... .......... .......... .......... 50% 24.6M 3s 2024-09-16T12:22:56.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 84850K .......... .......... .......... .......... .......... 50% 36.5M 3s 2024-09-16T12:22:56.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 84900K .......... .......... .......... .......... .......... 50% 25.6M 3s 2024-09-16T12:22:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 84950K .......... .......... .......... .......... .......... 50% 42.1M 3s 2024-09-16T12:22:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 85000K .......... .......... .......... .......... .......... 50% 30.7M 3s 2024-09-16T12:22:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 85050K .......... .......... .......... .......... .......... 50% 27.6M 3s 2024-09-16T12:22:56.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 85100K .......... .......... .......... .......... .......... 50% 24.3M 3s 2024-09-16T12:22:56.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 85150K .......... .......... .......... .......... .......... 50% 23.7M 3s 2024-09-16T12:22:56.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 85200K .......... .......... .......... .......... .......... 50% 26.1M 3s 2024-09-16T12:22:56.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 85250K .......... .......... .......... .......... .......... 50% 45.7M 3s 2024-09-16T12:22:56.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 85300K .......... .......... .......... .......... .......... 50% 24.5M 3s 2024-09-16T12:22:56.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 85350K .......... .......... .......... .......... .......... 50% 35.2M 3s 2024-09-16T12:22:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 85400K .......... .......... .......... .......... .......... 50% 25.2M 3s 2024-09-16T12:22:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 85450K .......... .......... .......... .......... .......... 50% 52.9M 3s 2024-09-16T12:22:56.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 85500K .......... .......... .......... .......... .......... 50% 23.8M 3s 2024-09-16T12:22:56.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 85550K .......... .......... .......... .......... .......... 50% 23.2M 3s 2024-09-16T12:22:56.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 85600K .......... .......... .......... .......... .......... 50% 32.7M 3s 2024-09-16T12:22:56.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 85650K .......... .......... .......... .......... .......... 50% 24.7M 3s 2024-09-16T12:22:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 85700K .......... .......... .......... .......... .......... 50% 25.6M 3s 2024-09-16T12:22:56.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 85750K .......... .......... .......... .......... .......... 50% 46.9M 3s 2024-09-16T12:22:56.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 85800K .......... .......... .......... .......... .......... 50% 24.7M 3s 2024-09-16T12:22:56.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 85850K .......... .......... .......... .......... .......... 50% 38.8M 3s 2024-09-16T12:22:56.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 85900K .......... .......... .......... .......... .......... 50% 24.8M 3s 2024-09-16T12:22:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 85950K .......... .......... .......... .......... .......... 51% 23.6M 3s 2024-09-16T12:22:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 86000K .......... .......... .......... .......... .......... 51% 47.2M 3s 2024-09-16T12:22:56.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 86050K .......... .......... .......... .......... .......... 51% 24.3M 3s 2024-09-16T12:22:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 86100K .......... .......... .......... .......... .......... 51% 36.5M 3s 2024-09-16T12:22:56.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 86150K .......... .......... .......... .......... .......... 51% 23.1M 3s 2024-09-16T12:22:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 86200K .......... .......... .......... .......... .......... 51% 25.4M 3s 2024-09-16T12:22:56.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 86250K .......... .......... .......... .......... .......... 51% 44.0M 3s 2024-09-16T12:22:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 86300K .......... .......... .......... .......... .......... 51% 25.0M 3s 2024-09-16T12:22:56.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 86350K .......... .......... .......... .......... .......... 51% 24.2M 3s 2024-09-16T12:22:56.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 86400K .......... .......... .......... .......... .......... 51% 39.4M 3s 2024-09-16T12:22:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 86450K .......... .......... .......... .......... .......... 51% 28.9M 3s 2024-09-16T12:22:56.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 86500K .......... .......... .......... .......... .......... 51% 33.9M 3s 2024-09-16T12:22:56.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 86550K .......... .......... .......... .......... .......... 51% 25.3M 3s 2024-09-16T12:22:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 86600K .......... .......... .......... .......... .......... 51% 22.5M 3s 2024-09-16T12:22:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 86650K .......... .......... .......... .......... .......... 51% 49.8M 3s 2024-09-16T12:22:56.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 86700K .......... .......... .......... .......... .......... 51% 23.3M 3s 2024-09-16T12:22:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 86750K .......... .......... .......... .......... .......... 51% 24.2M 3s 2024-09-16T12:22:56.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 86800K .......... .......... .......... .......... .......... 51% 46.2M 3s 2024-09-16T12:22:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 86850K .......... .......... .......... .......... .......... 51% 24.3M 3s 2024-09-16T12:22:56.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 86900K .......... .......... .......... .......... .......... 51% 38.1M 3s 2024-09-16T12:22:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 86950K .......... .......... .......... .......... .......... 51% 24.6M 3s 2024-09-16T12:22:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 87000K .......... .......... .......... .......... .......... 51% 39.7M 3s 2024-09-16T12:22:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 87050K .......... .......... .......... .......... .......... 51% 25.7M 3s 2024-09-16T12:22:56.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 87100K .......... .......... .......... .......... .......... 51% 41.3M 3s 2024-09-16T12:22:56.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 87150K .......... .......... .......... .......... .......... 51% 23.1M 3s 2024-09-16T12:22:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 87200K .......... .......... .......... .......... .......... 51% 21.8M 3s 2024-09-16T12:22:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 87250K .......... .......... .......... .......... .......... 51% 26.7M 3s 2024-09-16T12:22:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 87300K .......... .......... .......... .......... .......... 51% 52.6M 3s 2024-09-16T12:22:56.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 87350K .......... .......... .......... .......... .......... 51% 22.6M 3s 2024-09-16T12:22:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 87400K .......... .......... .......... .......... .......... 51% 38.3M 3s 2024-09-16T12:22:56.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 87450K .......... .......... .......... .......... .......... 51% 45.7M 3s 2024-09-16T12:22:56.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 87500K .......... .......... .......... .......... .......... 51% 25.8M 3s 2024-09-16T12:22:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 87550K .......... .......... .......... .......... .......... 51% 25.3M 3s 2024-09-16T12:22:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 87600K .......... .......... .......... .......... .......... 51% 22.7M 3s 2024-09-16T12:22:56.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 87650K .......... .......... .......... .......... .......... 52% 29.2M 3s 2024-09-16T12:22:56.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 87700K .......... .......... .......... .......... .......... 52% 30.1M 3s 2024-09-16T12:22:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 87750K .......... .......... .......... .......... .......... 52% 43.8M 3s 2024-09-16T12:22:56.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 87800K .......... .......... .......... .......... .......... 52% 24.7M 3s 2024-09-16T12:22:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 87850K .......... .......... .......... .......... .......... 52% 30.1M 3s 2024-09-16T12:22:56.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 87900K .......... .......... .......... .......... .......... 52% 22.6M 3s 2024-09-16T12:22:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 87950K .......... .......... .......... .......... .......... 52% 26.0M 3s 2024-09-16T12:22:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 88000K .......... .......... .......... .......... .......... 52% 26.2M 3s 2024-09-16T12:22:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 88050K .......... .......... .......... .......... .......... 52% 47.6M 3s 2024-09-16T12:22:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 88100K .......... .......... .......... .......... .......... 52% 25.3M 3s 2024-09-16T12:22:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 88150K .......... .......... .......... .......... .......... 52% 36.3M 3s 2024-09-16T12:22:56.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 88200K .......... .......... .......... .......... .......... 52% 22.5M 3s 2024-09-16T12:22:56.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 88250K .......... .......... .......... .......... .......... 52% 49.5M 3s 2024-09-16T12:22:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 88300K .......... .......... .......... .......... .......... 52% 24.6M 3s 2024-09-16T12:22:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 88350K .......... .......... .......... .......... .......... 52% 24.3M 3s 2024-09-16T12:22:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 88400K .......... .......... .......... .......... .......... 52% 33.6M 3s 2024-09-16T12:22:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 88450K .......... .......... .......... .......... .......... 52% 20.6M 3s 2024-09-16T12:22:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 88500K .......... .......... .......... .......... .......... 52% 32.0M 3s 2024-09-16T12:22:56.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 88550K .......... .......... .......... .......... .......... 52% 44.2M 3s 2024-09-16T12:22:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 88600K .......... .......... .......... .......... .......... 52% 27.7M 3s 2024-09-16T12:22:56.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 88650K .......... .......... .......... .......... .......... 52% 32.9M 3s 2024-09-16T12:22:56.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 88700K .......... .......... .......... .......... .......... 52% 24.8M 3s 2024-09-16T12:22:56.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 88750K .......... .......... .......... .......... .......... 52% 24.0M 3s 2024-09-16T12:22:56.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 88800K .......... .......... .......... .......... .......... 52% 44.4M 3s 2024-09-16T12:22:56.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 88850K .......... .......... .......... .......... .......... 52% 31.1M 3s 2024-09-16T12:22:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 88900K .......... .......... .......... .......... .......... 52% 19.8M 3s 2024-09-16T12:22:56.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 88950K .......... .......... .......... .......... .......... 52% 36.8M 3s 2024-09-16T12:22:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 89000K .......... .......... .......... .......... .......... 52% 24.8M 3s 2024-09-16T12:22:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 89050K .......... .......... .......... .......... .......... 52% 53.0M 3s 2024-09-16T12:22:56.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 89100K .......... .......... .......... .......... .......... 52% 23.0M 3s 2024-09-16T12:22:56.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 89150K .......... .......... .......... .......... .......... 52% 25.1M 3s 2024-09-16T12:22:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 89200K .......... .......... .......... .......... .......... 52% 35.7M 3s 2024-09-16T12:22:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 89250K .......... .......... .......... .......... .......... 52% 24.0M 3s 2024-09-16T12:22:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 89300K .......... .......... .......... .......... .......... 53% 50.8M 3s 2024-09-16T12:22:56.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 89350K .......... .......... .......... .......... .......... 53% 25.6M 3s 2024-09-16T12:22:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 89400K .......... .......... .......... .......... .......... 53% 33.6M 3s 2024-09-16T12:22:56.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 89450K .......... .......... .......... .......... .......... 53% 25.0M 3s 2024-09-16T12:22:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 89500K .......... .......... .......... .......... .......... 53% 22.3M 3s 2024-09-16T12:22:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 89550K .......... .......... .......... .......... .......... 53% 25.3M 3s 2024-09-16T12:22:56.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 89600K .......... .......... .......... .......... .......... 53% 51.6M 3s 2024-09-16T12:22:56.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 89650K .......... .......... .......... .......... .......... 53% 23.9M 3s 2024-09-16T12:22:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 89700K .......... .......... .......... .......... .......... 53% 35.7M 3s 2024-09-16T12:22:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 89750K .......... .......... .......... .......... .......... 53% 24.4M 3s 2024-09-16T12:22:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 89800K .......... .......... .......... .......... .......... 53% 46.9M 3s 2024-09-16T12:22:56.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 89850K .......... .......... .......... .......... .......... 53% 25.2M 3s 2024-09-16T12:22:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 89900K .......... .......... .......... .......... .......... 53% 36.7M 3s 2024-09-16T12:22:56.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 89950K .......... .......... .......... .......... .......... 53% 24.2M 3s 2024-09-16T12:22:56.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 90000K .......... .......... .......... .......... .......... 53% 25.9M 3s 2024-09-16T12:22:56.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 90050K .......... .......... .......... .......... .......... 53% 44.0M 3s 2024-09-16T12:22:56.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 90100K .......... .......... .......... .......... .......... 53% 25.7M 3s 2024-09-16T12:22:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 90150K .......... .......... .......... .......... .......... 53% 33.8M 3s 2024-09-16T12:22:56.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 90200K .......... .......... .......... .......... .......... 53% 23.1M 3s 2024-09-16T12:22:56.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 90250K .......... .......... .......... .......... .......... 53% 25.8M 3s 2024-09-16T12:22:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 90300K .......... .......... .......... .......... .......... 53% 45.2M 3s 2024-09-16T12:22:56.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 90350K .......... .......... .......... .......... .......... 53% 25.9M 3s 2024-09-16T12:22:56.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 90400K .......... .......... .......... .......... .......... 53% 23.3M 3s 2024-09-16T12:22:56.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 90450K .......... .......... .......... .......... .......... 53% 37.6M 3s 2024-09-16T12:22:56.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 90500K .......... .......... .......... .......... .......... 53% 25.0M 3s 2024-09-16T12:22:56.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 90550K .......... .......... .......... .......... .......... 53% 42.0M 3s 2024-09-16T12:22:56.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 90600K .......... .......... .......... .......... .......... 53% 25.5M 3s 2024-09-16T12:22:56.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 90650K .......... .......... .......... .......... .......... 53% 39.0M 3s 2024-09-16T12:22:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 90700K .......... .......... .......... .......... .......... 53% 24.2M 3s 2024-09-16T12:22:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 90750K .......... .......... .......... .......... .......... 53% 26.8M 3s 2024-09-16T12:22:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 90800K .......... .......... .......... .......... .......... 53% 24.6M 3s 2024-09-16T12:22:56.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 90850K .......... .......... .......... .......... .......... 53% 37.1M 3s 2024-09-16T12:22:56.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 90900K .......... .......... .......... .......... .......... 53% 25.5M 3s 2024-09-16T12:22:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 90950K .......... .......... .......... .......... .......... 53% 41.3M 3s 2024-09-16T12:22:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 91000K .......... .......... .......... .......... .......... 54% 22.4M 3s 2024-09-16T12:22:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 91050K .......... .......... .......... .......... .......... 54% 46.6M 3s 2024-09-16T12:22:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 91100K .......... .......... .......... .......... .......... 54% 25.2M 3s 2024-09-16T12:22:56.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 91150K .......... .......... .......... .......... .......... 54% 24.0M 3s 2024-09-16T12:22:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 91200K .......... .......... .......... .......... .......... 54% 35.3M 3s 2024-09-16T12:22:56.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 91250K .......... .......... .......... .......... .......... 54% 23.3M 3s 2024-09-16T12:22:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 91300K .......... .......... .......... .......... .......... 54% 25.9M 3s 2024-09-16T12:22:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 91350K .......... .......... .......... .......... .......... 54% 47.7M 3s 2024-09-16T12:22:56.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 91400K .......... .......... .......... .......... .......... 54% 23.1M 3s 2024-09-16T12:22:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 91450K .......... .......... .......... .......... .......... 54% 40.0M 3s 2024-09-16T12:22:56.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 91500K .......... .......... .......... .......... .......... 54% 24.1M 3s 2024-09-16T12:22:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 91550K .......... .......... .......... .......... .......... 54% 24.0M 3s 2024-09-16T12:22:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 91600K .......... .......... .......... .......... .......... 54% 46.3M 3s 2024-09-16T12:22:56.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 91650K .......... .......... .......... .......... .......... 54% 25.1M 3s 2024-09-16T12:22:56.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 91700K .......... .......... .......... .......... .......... 54% 36.0M 3s 2024-09-16T12:22:56.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 91750K .......... .......... .......... .......... .......... 54% 23.8M 3s 2024-09-16T12:22:56.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 91800K .......... .......... .......... .......... .......... 54% 23.8M 3s 2024-09-16T12:22:56.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 91850K .......... .......... .......... .......... .......... 54% 71.4M 3s 2024-09-16T12:22:56.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 91900K .......... .......... .......... .......... .......... 54% 30.7M 3s 2024-09-16T12:22:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 91950K .......... .......... .......... .......... .......... 54% 17.8M 3s 2024-09-16T12:22:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 92000K .......... .......... .......... .......... .......... 54% 36.6M 3s 2024-09-16T12:22:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 92050K .......... .......... .......... .......... .......... 54% 23.9M 3s 2024-09-16T12:22:56.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 92100K .......... .......... .......... .......... .......... 54% 48.4M 3s 2024-09-16T12:22:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 92150K .......... .......... .......... .......... .......... 54% 24.4M 3s 2024-09-16T12:22:56.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 92200K .......... .......... .......... .......... .......... 54% 28.6M 3s 2024-09-16T12:22:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 92250K .......... .......... .......... .......... .......... 54% 31.0M 3s 2024-09-16T12:22:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 92300K .......... .......... .......... .......... .......... 54% 49.5M 3s 2024-09-16T12:22:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 92350K .......... .......... .......... .......... .......... 54% 15.8M 3s 2024-09-16T12:22:56.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 92400K .......... .......... .......... .......... .......... 54% 49.1M 3s 2024-09-16T12:22:56.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 92450K .......... .......... .......... .......... .......... 54% 34.6M 3s 2024-09-16T12:22:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 92500K .......... .......... .......... .......... .......... 54% 24.4M 3s 2024-09-16T12:22:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 92550K .......... .......... .......... .......... .......... 54% 25.3M 3s 2024-09-16T12:22:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 92600K .......... .......... .......... .......... .......... 54% 46.8M 3s 2024-09-16T12:22:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 92650K .......... .......... .......... .......... .......... 54% 28.9M 3s 2024-09-16T12:22:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 92700K .......... .......... .......... .......... .......... 55% 31.4M 3s 2024-09-16T12:22:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 92750K .......... .......... .......... .......... .......... 55% 21.3M 3s 2024-09-16T12:22:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 92800K .......... .......... .......... .......... .......... 55% 28.3M 3s 2024-09-16T12:22:56.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 92850K .......... .......... .......... .......... .......... 55% 56.1M 3s 2024-09-16T12:22:56.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 92900K .......... .......... .......... .......... .......... 55% 22.6M 3s 2024-09-16T12:22:56.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 92950K .......... .......... .......... .......... .......... 55% 31.9M 3s 2024-09-16T12:22:56.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 93000K .......... .......... .......... .......... .......... 55% 25.9M 3s 2024-09-16T12:22:56.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 93050K .......... .......... .......... .......... .......... 55% 23.9M 3s 2024-09-16T12:22:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 93100K .......... .......... .......... .......... .......... 55% 48.3M 3s 2024-09-16T12:22:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 93150K .......... .......... .......... .......... .......... 55% 23.7M 3s 2024-09-16T12:22:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 93200K .......... .......... .......... .......... .......... 55% 25.8M 3s 2024-09-16T12:22:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 93250K .......... .......... .......... .......... .......... 55% 36.2M 3s 2024-09-16T12:22:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 93300K .......... .......... .......... .......... .......... 55% 24.9M 3s 2024-09-16T12:22:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 93350K .......... .......... .......... .......... .......... 55% 46.6M 3s 2024-09-16T12:22:56.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 93400K .......... .......... .......... .......... .......... 55% 23.9M 3s 2024-09-16T12:22:56.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 93450K .......... .......... .......... .......... .......... 55% 40.1M 3s 2024-09-16T12:22:56.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 93500K .......... .......... .......... .......... .......... 55% 21.7M 3s 2024-09-16T12:22:56.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 93550K .......... .......... .......... .......... .......... 55% 23.7M 3s 2024-09-16T12:22:56.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 93600K .......... .......... .......... .......... .......... 55% 25.3M 3s 2024-09-16T12:22:56.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 93650K .......... .......... .......... .......... .......... 55% 54.9M 3s 2024-09-16T12:22:56.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 93700K .......... .......... .......... .......... .......... 55% 23.4M 3s 2024-09-16T12:22:56.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 93750K .......... .......... .......... .......... .......... 55% 36.1M 3s 2024-09-16T12:22:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 93800K .......... .......... .......... .......... .......... 55% 24.8M 3s 2024-09-16T12:22:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 93850K .......... .......... .......... .......... .......... 55% 46.0M 3s 2024-09-16T12:22:56.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 93900K .......... .......... .......... .......... .......... 55% 25.4M 3s 2024-09-16T12:22:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 93950K .......... .......... .......... .......... .......... 55% 24.7M 3s 2024-09-16T12:22:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 94000K .......... .......... .......... .......... .......... 55% 34.2M 3s 2024-09-16T12:22:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 94050K .......... .......... .......... .......... .......... 55% 23.4M 3s 2024-09-16T12:22:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 94100K .......... .......... .......... .......... .......... 55% 25.5M 3s 2024-09-16T12:22:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 94150K .......... .......... .......... .......... .......... 55% 55.4M 3s 2024-09-16T12:22:56.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 94200K .......... .......... .......... .......... .......... 55% 22.6M 3s 2024-09-16T12:22:56.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 94250K .......... .......... .......... .......... .......... 55% 37.4M 3s 2024-09-16T12:22:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 94300K .......... .......... .......... .......... .......... 55% 24.5M 3s 2024-09-16T12:22:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 94350K .......... .......... .......... .......... .......... 56% 25.9M 3s 2024-09-16T12:22:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 94400K .......... .......... .......... .......... .......... 56% 40.8M 3s 2024-09-16T12:22:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 94450K .......... .......... .......... .......... .......... 56% 25.3M 3s 2024-09-16T12:22:56.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 94500K .......... .......... .......... .......... .......... 56% 36.5M 3s 2024-09-16T12:22:56.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 94550K .......... .......... .......... .......... .......... 56% 25.0M 3s 2024-09-16T12:22:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 94600K .......... .......... .......... .......... .......... 56% 22.2M 3s 2024-09-16T12:22:56.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 94650K .......... .......... .......... .......... .......... 56% 61.0M 3s 2024-09-16T12:22:56.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 94700K .......... .......... .......... .......... .......... 56% 34.4M 3s 2024-09-16T12:22:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 94750K .......... .......... .......... .......... .......... 56% 10.2M 3s 2024-09-16T12:22:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 94800K .......... .......... .......... .......... .......... 56% 146M 3s 2024-09-16T12:22:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 94850K .......... .......... .......... .......... .......... 56% 44.1M 3s 2024-09-16T12:22:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 94900K .......... .......... .......... .......... .......... 56% 58.9M 3s 2024-09-16T12:22:56.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 94950K .......... .......... .......... .......... .......... 56% 24.5M 3s 2024-09-16T12:22:56.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 95000K .......... .......... .......... .......... .......... 56% 35.4M 3s 2024-09-16T12:22:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 95050K .......... .......... .......... .......... .......... 56% 24.5M 3s 2024-09-16T12:22:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 95100K .......... .......... .......... .......... .......... 56% 54.9M 3s 2024-09-16T12:22:56.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 95150K .......... .......... .......... .......... .......... 56% 12.9M 3s 2024-09-16T12:22:56.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 95200K .......... .......... .......... .......... .......... 56% 56.6M 3s 2024-09-16T12:22:56.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 95250K .......... .......... .......... .......... .......... 56% 36.5M 3s 2024-09-16T12:22:56.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 95300K .......... .......... .......... .......... .......... 56% 23.1M 3s 2024-09-16T12:22:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 95350K .......... .......... .......... .......... .......... 56% 24.2M 3s 2024-09-16T12:22:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 95400K .......... .......... .......... .......... .......... 56% 47.2M 3s 2024-09-16T12:22:56.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 95450K .......... .......... .......... .......... .......... 56% 25.7M 3s 2024-09-16T12:22:56.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 95500K .......... .......... .......... .......... .......... 56% 36.2M 3s 2024-09-16T12:22:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 95550K .......... .......... .......... .......... .......... 56% 19.2M 3s 2024-09-16T12:22:56.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 95600K .......... .......... .......... .......... .......... 56% 25.7M 3s 2024-09-16T12:22:56.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 95650K .......... .......... .......... .......... .......... 56% 46.4M 3s 2024-09-16T12:22:56.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 95700K .......... .......... .......... .......... .......... 56% 25.4M 3s 2024-09-16T12:22:56.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 95750K .......... .......... .......... .......... .......... 56% 31.5M 3s 2024-09-16T12:22:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 95800K .......... .......... .......... .......... .......... 56% 25.1M 3s 2024-09-16T12:22:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 95850K .......... .......... .......... .......... .......... 56% 25.0M 3s 2024-09-16T12:22:56.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 95900K .......... .......... .......... .......... .......... 56% 48.5M 3s 2024-09-16T12:22:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 95950K .......... .......... .......... .......... .......... 56% 24.2M 3s 2024-09-16T12:22:57.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 96000K .......... .......... .......... .......... .......... 56% 25.2M 3s 2024-09-16T12:22:57.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 96050K .......... .......... .......... .......... .......... 57% 44.3M 3s 2024-09-16T12:22:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 96100K .......... .......... .......... .......... .......... 57% 23.5M 3s 2024-09-16T12:22:57.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 96150K .......... .......... .......... .......... .......... 57% 34.4M 3s 2024-09-16T12:22:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 96200K .......... .......... .......... .......... .......... 57% 23.3M 3s 2024-09-16T12:22:57.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 96250K .......... .......... .......... .......... .......... 57% 47.1M 2s 2024-09-16T12:22:57.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 96300K .......... .......... .......... .......... .......... 57% 26.8M 2s 2024-09-16T12:22:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 96350K .......... .......... .......... .......... .......... 57% 20.2M 2s 2024-09-16T12:22:57.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 96400K .......... .......... .......... .......... .......... 57% 25.5M 2s 2024-09-16T12:22:57.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 96450K .......... .......... .......... .......... .......... 57% 44.1M 2s 2024-09-16T12:22:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 96500K .......... .......... .......... .......... .......... 57% 33.6M 2s 2024-09-16T12:22:57.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 96550K .......... .......... .......... .......... .......... 57% 21.5M 2s 2024-09-16T12:22:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 96600K .......... .......... .......... .......... .......... 57% 31.5M 2s 2024-09-16T12:22:57.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 96650K .......... .......... .......... .......... .......... 57% 47.6M 2s 2024-09-16T12:22:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 96700K .......... .......... .......... .......... .......... 57% 27.4M 2s 2024-09-16T12:22:57.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 96750K .......... .......... .......... .......... .......... 57% 22.2M 2s 2024-09-16T12:22:57.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 96800K .......... .......... .......... .......... .......... 57% 34.8M 2s 2024-09-16T12:22:57.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 96850K .......... .......... .......... .......... .......... 57% 20.9M 2s 2024-09-16T12:22:57.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 96900K .......... .......... .......... .......... .......... 57% 49.9M 2s 2024-09-16T12:22:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 96950K .......... .......... .......... .......... .......... 57% 29.7M 2s 2024-09-16T12:22:57.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 97000K .......... .......... .......... .......... .......... 57% 23.7M 2s 2024-09-16T12:22:57.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 97050K .......... .......... .......... .......... .......... 57% 33.0M 2s 2024-09-16T12:22:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 97100K .......... .......... .......... .......... .......... 57% 28.7M 2s 2024-09-16T12:22:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 97150K .......... .......... .......... .......... .......... 57% 22.8M 2s 2024-09-16T12:22:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 97200K .......... .......... .......... .......... .......... 57% 46.0M 2s 2024-09-16T12:22:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 97250K .......... .......... .......... .......... .......... 57% 25.3M 2s 2024-09-16T12:22:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 97300K .......... .......... .......... .......... .......... 57% 37.6M 2s 2024-09-16T12:22:57.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 97350K .......... .......... .......... .......... .......... 57% 24.5M 2s 2024-09-16T12:22:57.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 97400K .......... .......... .......... .......... .......... 57% 47.7M 2s 2024-09-16T12:22:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 97450K .......... .......... .......... .......... .......... 57% 23.9M 2s 2024-09-16T12:22:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 97500K .......... .......... .......... .......... .......... 57% 35.4M 2s 2024-09-16T12:22:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 97550K .......... .......... .......... .......... .......... 57% 18.4M 2s 2024-09-16T12:22:57.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 97600K .......... .......... .......... .......... .......... 57% 39.1M 2s 2024-09-16T12:22:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 97650K .......... .......... .......... .......... .......... 57% 24.4M 2s 2024-09-16T12:22:57.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 97700K .......... .......... .......... .......... .......... 57% 44.4M 2s 2024-09-16T12:22:57.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 97750K .......... .......... .......... .......... .......... 58% 24.1M 2s 2024-09-16T12:22:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 97800K .......... .......... .......... .......... .......... 58% 28.4M 2s 2024-09-16T12:22:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 97850K .......... .......... .......... .......... .......... 58% 32.8M 2s 2024-09-16T12:22:57.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 97900K .......... .......... .......... .......... .......... 58% 47.4M 2s 2024-09-16T12:22:57.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 97950K .......... .......... .......... .......... .......... 58% 23.0M 2s 2024-09-16T12:22:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 98000K .......... .......... .......... .......... .......... 58% 25.1M 2s 2024-09-16T12:22:57.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 98050K .......... .......... .......... .......... .......... 58% 36.8M 2s 2024-09-16T12:22:57.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 98100K .......... .......... .......... .......... .......... 58% 23.7M 2s 2024-09-16T12:22:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 98150K .......... .......... .......... .......... .......... 58% 24.1M 2s 2024-09-16T12:22:57.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 98200K .......... .......... .......... .......... .......... 58% 46.1M 2s 2024-09-16T12:22:57.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 98250K .......... .......... .......... .......... .......... 58% 29.2M 2s 2024-09-16T12:22:57.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 98300K .......... .......... .......... .......... .......... 58% 33.2M 2s 2024-09-16T12:22:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 98350K .......... .......... .......... .......... .......... 58% 23.4M 2s 2024-09-16T12:22:57.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 98400K .......... .......... .......... .......... .......... 58% 24.5M 2s 2024-09-16T12:22:57.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 98450K .......... .......... .......... .......... .......... 58% 45.6M 2s 2024-09-16T12:22:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 98500K .......... .......... .......... .......... .......... 58% 25.4M 2s 2024-09-16T12:22:57.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 98550K .......... .......... .......... .......... .......... 58% 36.1M 2s 2024-09-16T12:22:57.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 98600K .......... .......... .......... .......... .......... 58% 22.9M 2s 2024-09-16T12:22:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 98650K .......... .......... .......... .......... .......... 58% 23.7M 2s 2024-09-16T12:22:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 98700K .......... .......... .......... .......... .......... 58% 52.4M 2s 2024-09-16T12:22:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 98750K .......... .......... .......... .......... .......... 58% 23.1M 2s 2024-09-16T12:22:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 98800K .......... .......... .......... .......... .......... 58% 22.8M 2s 2024-09-16T12:22:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 98850K .......... .......... .......... .......... .......... 58% 30.7M 2s 2024-09-16T12:22:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 98900K .......... .......... .......... .......... .......... 58% 25.9M 2s 2024-09-16T12:22:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 98950K .......... .......... .......... .......... .......... 58% 39.3M 2s 2024-09-16T12:22:57.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 99000K .......... .......... .......... .......... .......... 58% 25.4M 2s 2024-09-16T12:22:57.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 99050K .......... .......... .......... .......... .......... 58% 39.1M 2s 2024-09-16T12:22:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 99100K .......... .......... .......... .......... .......... 58% 24.5M 2s 2024-09-16T12:22:57.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 99150K .......... .......... .......... .......... .......... 58% 21.9M 2s 2024-09-16T12:22:57.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 99200K .......... .......... .......... .......... .......... 58% 20.9M 2s 2024-09-16T12:22:57.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 99250K .......... .......... .......... .......... .......... 58% 45.1M 2s 2024-09-16T12:22:57.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 99300K .......... .......... .......... .......... .......... 58% 25.2M 2s 2024-09-16T12:22:57.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 99350K .......... .......... .......... .......... .......... 58% 25.7M 2s 2024-09-16T12:22:57.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 99400K .......... .......... .......... .......... .......... 58% 34.3M 2s 2024-09-16T12:22:57.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 99450K .......... .......... .......... .......... .......... 59% 46.3M 2s 2024-09-16T12:22:57.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 99500K .......... .......... .......... .......... .......... 59% 24.2M 2s 2024-09-16T12:22:57.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 99550K .......... .......... .......... .......... .......... 59% 24.5M 2s 2024-09-16T12:22:57.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 99600K .......... .......... .......... .......... .......... 59% 37.9M 2s 2024-09-16T12:22:57.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 99650K .......... .......... .......... .......... .......... 59% 25.4M 2s 2024-09-16T12:22:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 99700K .......... .......... .......... .......... .......... 59% 22.7M 2s 2024-09-16T12:22:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 99750K .......... .......... .......... .......... .......... 59% 51.7M 2s 2024-09-16T12:22:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 99800K .......... .......... .......... .......... .......... 59% 23.5M 2s 2024-09-16T12:22:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 99850K .......... .......... .......... .......... .......... 59% 47.2M 2s 2024-09-16T12:22:57.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 99900K .......... .......... .......... .......... .......... 59% 21.6M 2s 2024-09-16T12:22:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 99950K .......... .......... .......... .......... .......... 59% 24.7M 2s 2024-09-16T12:22:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:100000K .......... .......... .......... .......... .......... 59% 45.0M 2s 2024-09-16T12:22:57.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:100050K .......... .......... .......... .......... .......... 59% 25.2M 2s 2024-09-16T12:22:57.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:100100K .......... .......... .......... .......... .......... 59% 33.2M 2s 2024-09-16T12:22:57.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:100150K .......... .......... .......... .......... .......... 59% 26.2M 2s 2024-09-16T12:22:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:100200K .......... .......... .......... .......... .......... 59% 49.8M 2s 2024-09-16T12:22:57.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:100250K .......... .......... .......... .......... .......... 59% 24.4M 2s 2024-09-16T12:22:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:100300K .......... .......... .......... .......... .......... 59% 32.0M 2s 2024-09-16T12:22:57.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:100350K .......... .......... .......... .......... .......... 59% 24.9M 2s 2024-09-16T12:22:57.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:100400K .......... .......... .......... .......... .......... 59% 25.4M 2s 2024-09-16T12:22:57.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:100450K .......... .......... .......... .......... .......... 59% 23.9M 2s 2024-09-16T12:22:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:100500K .......... .......... .......... .......... .......... 59% 49.4M 2s 2024-09-16T12:22:57.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:100550K .......... .......... .......... .......... .......... 59% 24.8M 2s 2024-09-16T12:22:57.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:100600K .......... .......... .......... .......... .......... 59% 32.7M 2s 2024-09-16T12:22:57.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:100650K .......... .......... .......... .......... .......... 59% 26.7M 2s 2024-09-16T12:22:57.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:100700K .......... .......... .......... .......... .......... 59% 47.2M 2s 2024-09-16T12:22:57.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:100750K .......... .......... .......... .......... .......... 59% 24.5M 2s 2024-09-16T12:22:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:100800K .......... .......... .......... .......... .......... 59% 23.8M 2s 2024-09-16T12:22:57.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:100850K .......... .......... .......... .......... .......... 59% 36.7M 2s 2024-09-16T12:22:57.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:100900K .......... .......... .......... .......... .......... 59% 23.4M 2s 2024-09-16T12:22:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:100950K .......... .......... .......... .......... .......... 59% 24.7M 2s 2024-09-16T12:22:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:101000K .......... .......... .......... .......... .......... 59% 49.0M 2s 2024-09-16T12:22:57.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:101050K .......... .......... .......... .......... .......... 59% 23.4M 2s 2024-09-16T12:22:57.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:101100K .......... .......... .......... .......... .......... 60% 41.9M 2s 2024-09-16T12:22:57.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:101150K .......... .......... .......... .......... .......... 60% 25.1M 2s 2024-09-16T12:22:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:101200K .......... .......... .......... .......... .......... 60% 22.9M 2s 2024-09-16T12:22:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:101250K .......... .......... .......... .......... .......... 60% 47.7M 2s 2024-09-16T12:22:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:101300K .......... .......... .......... .......... .......... 60% 24.3M 2s 2024-09-16T12:22:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:101350K .......... .......... .......... .......... .......... 60% 35.4M 2s 2024-09-16T12:22:57.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:101400K .......... .......... .......... .......... .......... 60% 21.9M 2s 2024-09-16T12:22:57.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:101450K .......... .......... .......... .......... .......... 60% 27.4M 2s 2024-09-16T12:22:57.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:101500K .......... .......... .......... .......... .......... 60% 51.2M 2s 2024-09-16T12:22:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:101550K .......... .......... .......... .......... .......... 60% 22.4M 2s 2024-09-16T12:22:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:101600K .......... .......... .......... .......... .......... 60% 21.3M 2s 2024-09-16T12:22:57.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:101650K .......... .......... .......... .......... .......... 60% 51.3M 2s 2024-09-16T12:22:57.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:101700K .......... .......... .......... .......... .......... 60% 24.8M 2s 2024-09-16T12:22:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:101750K .......... .......... .......... .......... .......... 60% 42.5M 2s 2024-09-16T12:22:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:101800K .......... .......... .......... .......... .......... 60% 23.3M 2s 2024-09-16T12:22:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:101850K .......... .......... .......... .......... .......... 60% 41.6M 2s 2024-09-16T12:22:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:101900K .......... .......... .......... .......... .......... 60% 25.4M 2s 2024-09-16T12:22:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:101950K .......... .......... .......... .......... .......... 60% 24.0M 2s 2024-09-16T12:22:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:102000K .......... .......... .......... .......... .......... 60% 22.6M 2s 2024-09-16T12:22:57.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:102050K .......... .......... .......... .......... .......... 60% 54.8M 2s 2024-09-16T12:22:57.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:102100K .......... .......... .......... .......... .......... 60% 23.7M 2s 2024-09-16T12:22:57.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:102150K .......... .......... .......... .......... .......... 60% 36.4M 2s 2024-09-16T12:22:57.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:102200K .......... .......... .......... .......... .......... 60% 24.1M 2s 2024-09-16T12:22:57.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:102250K .......... .......... .......... .......... .......... 60% 48.0M 2s 2024-09-16T12:22:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:102300K .......... .......... .......... .......... .......... 60% 25.3M 2s 2024-09-16T12:22:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:102350K .......... .......... .......... .......... .......... 60% 23.8M 2s 2024-09-16T12:22:57.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:102400K .......... .......... .......... .......... .......... 60% 37.7M 2s 2024-09-16T12:22:57.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:102450K .......... .......... .......... .......... .......... 60% 24.8M 2s 2024-09-16T12:22:57.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:102500K .......... .......... .......... .......... .......... 60% 38.5M 2s 2024-09-16T12:22:57.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:102550K .......... .......... .......... .......... .......... 60% 23.9M 2s 2024-09-16T12:22:57.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:102600K .......... .......... .......... .......... .......... 60% 35.7M 2s 2024-09-16T12:22:57.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:102650K .......... .......... .......... .......... .......... 60% 21.9M 2s 2024-09-16T12:22:57.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:102700K .......... .......... .......... .......... .......... 60% 27.4M 2s 2024-09-16T12:22:57.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:102750K .......... .......... .......... .......... .......... 60% 23.7M 2s 2024-09-16T12:22:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:102800K .......... .......... .......... .......... .......... 61% 45.5M 2s 2024-09-16T12:22:57.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:102850K .......... .......... .......... .......... .......... 61% 24.9M 2s 2024-09-16T12:22:57.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:102900K .......... .......... .......... .......... .......... 61% 37.0M 2s 2024-09-16T12:22:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:102950K .......... .......... .......... .......... .......... 61% 20.4M 2s 2024-09-16T12:22:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:103000K .......... .......... .......... .......... .......... 61% 43.4M 2s 2024-09-16T12:22:57.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:103050K .......... .......... .......... .......... .......... 61% 26.7M 2s 2024-09-16T12:22:57.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:103100K .......... .......... .......... .......... .......... 61% 35.9M 2s 2024-09-16T12:22:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:103150K .......... .......... .......... .......... .......... 61% 22.1M 2s 2024-09-16T12:22:57.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:103200K .......... .......... .......... .......... .......... 61% 25.4M 2s 2024-09-16T12:22:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:103250K .......... .......... .......... .......... .......... 61% 25.3M 2s 2024-09-16T12:22:57.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:103300K .......... .......... .......... .......... .......... 61% 44.8M 2s 2024-09-16T12:22:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:103350K .......... .......... .......... .......... .......... 61% 24.4M 2s 2024-09-16T12:22:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:103400K .......... .......... .......... .......... .......... 61% 37.6M 2s 2024-09-16T12:22:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:103450K .......... .......... .......... .......... .......... 61% 24.6M 2s 2024-09-16T12:22:57.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:103500K .......... .......... .......... .......... .......... 61% 46.9M 2s 2024-09-16T12:22:57.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:103550K .......... .......... .......... .......... .......... 61% 24.2M 2s 2024-09-16T12:22:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:103600K .......... .......... .......... .......... .......... 61% 25.3M 2s 2024-09-16T12:22:57.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:103650K .......... .......... .......... .......... .......... 61% 36.9M 2s 2024-09-16T12:22:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:103700K .......... .......... .......... .......... .......... 61% 23.7M 2s 2024-09-16T12:22:57.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:103750K .......... .......... .......... .......... .......... 61% 24.1M 2s 2024-09-16T12:22:57.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:103800K .......... .......... .......... .......... .......... 61% 41.4M 2s 2024-09-16T12:22:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:103850K .......... .......... .......... .......... .......... 61% 26.8M 2s 2024-09-16T12:22:57.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:103900K .......... .......... .......... .......... .......... 61% 36.3M 2s 2024-09-16T12:22:57.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:103950K .......... .......... .......... .......... .......... 61% 24.0M 2s 2024-09-16T12:22:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:104000K .......... .......... .......... .......... .......... 61% 24.3M 2s 2024-09-16T12:22:57.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:104050K .......... .......... .......... .......... .......... 61% 52.3M 2s 2024-09-16T12:22:57.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:104100K .......... .......... .......... .......... .......... 61% 24.8M 2s 2024-09-16T12:22:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:104150K .......... .......... .......... .......... .......... 61% 33.0M 2s 2024-09-16T12:22:57.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:104200K .......... .......... .......... .......... .......... 61% 25.0M 2s 2024-09-16T12:22:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:104250K .......... .......... .......... .......... .......... 61% 23.5M 2s 2024-09-16T12:22:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:104300K .......... .......... .......... .......... .......... 61% 50.4M 2s 2024-09-16T12:22:57.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:104350K .......... .......... .......... .......... .......... 61% 25.8M 2s 2024-09-16T12:22:57.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:104400K .......... .......... .......... .......... .......... 61% 23.2M 2s 2024-09-16T12:22:57.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:104450K .......... .......... .......... .......... .......... 61% 36.7M 2s 2024-09-16T12:22:57.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:104500K .......... .......... .......... .......... .......... 62% 24.9M 2s 2024-09-16T12:22:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:104550K .......... .......... .......... .......... .......... 62% 43.8M 2s 2024-09-16T12:22:57.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:104600K .......... .......... .......... .......... .......... 62% 24.0M 2s 2024-09-16T12:22:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:104650K .......... .......... .......... .......... .......... 62% 40.9M 2s 2024-09-16T12:22:57.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:104700K .......... .......... .......... .......... .......... 62% 25.1M 2s 2024-09-16T12:22:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:104750K .......... .......... .......... .......... .......... 62% 23.8M 2s 2024-09-16T12:22:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:104800K .......... .......... .......... .......... .......... 62% 22.4M 2s 2024-09-16T12:22:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:104850K .......... .......... .......... .......... .......... 62% 52.6M 2s 2024-09-16T12:22:57.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:104900K .......... .......... .......... .......... .......... 62% 24.2M 2s 2024-09-16T12:22:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:104950K .......... .......... .......... .......... .......... 62% 35.2M 2s 2024-09-16T12:22:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:105000K .......... .......... .......... .......... .......... 62% 25.4M 2s 2024-09-16T12:22:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:105050K .......... .......... .......... .......... .......... 62% 43.7M 2s 2024-09-16T12:22:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:105100K .......... .......... .......... .......... .......... 62% 28.1M 2s 2024-09-16T12:22:57.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:105150K .......... .......... .......... .......... .......... 62% 21.6M 2s 2024-09-16T12:22:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:105200K .......... .......... .......... .......... .......... 62% 38.2M 2s 2024-09-16T12:22:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:105250K .......... .......... .......... .......... .......... 62% 25.6M 2s 2024-09-16T12:22:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:105300K .......... .......... .......... .......... .......... 62% 48.5M 2s 2024-09-16T12:22:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:105350K .......... .......... .......... .......... .......... 62% 22.2M 2s 2024-09-16T12:22:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:105400K .......... .......... .......... .......... .......... 62% 35.6M 2s 2024-09-16T12:22:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:105450K .......... .......... .......... .......... .......... 62% 24.6M 2s 2024-09-16T12:22:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:105500K .......... .......... .......... .......... .......... 62% 25.7M 2s 2024-09-16T12:22:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:105550K .......... .......... .......... .......... .......... 62% 24.4M 2s 2024-09-16T12:22:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:105600K .......... .......... .......... .......... .......... 62% 40.8M 2s 2024-09-16T12:22:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:105650K .......... .......... .......... .......... .......... 62% 26.7M 2s 2024-09-16T12:22:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:105700K .......... .......... .......... .......... .......... 62% 36.6M 2s 2024-09-16T12:22:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:105750K .......... .......... .......... .......... .......... 62% 24.3M 2s 2024-09-16T12:22:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:105800K .......... .......... .......... .......... .......... 62% 43.1M 2s 2024-09-16T12:22:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:105850K .......... .......... .......... .......... .......... 62% 25.7M 2s 2024-09-16T12:22:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:105900K .......... .......... .......... .......... .......... 62% 39.0M 2s 2024-09-16T12:22:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:105950K .......... .......... .......... .......... .......... 62% 21.9M 2s 2024-09-16T12:22:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:106000K .......... .......... .......... .......... .......... 62% 24.9M 2s 2024-09-16T12:22:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:106050K .......... .......... .......... .......... .......... 62% 25.7M 2s 2024-09-16T12:22:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:106100K .......... .......... .......... .......... .......... 62% 42.2M 2s 2024-09-16T12:22:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:106150K .......... .......... .......... .......... .......... 63% 25.0M 2s 2024-09-16T12:22:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:106200K .......... .......... .......... .......... .......... 63% 40.2M 2s 2024-09-16T12:22:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:106250K .......... .......... .......... .......... .......... 63% 24.2M 2s 2024-09-16T12:22:57.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:106300K .......... .......... .......... .......... .......... 63% 41.4M 2s 2024-09-16T12:22:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:106350K .......... .......... .......... .......... .......... 63% 24.6M 2s 2024-09-16T12:22:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:106400K .......... .......... .......... .......... .......... 63% 23.1M 2s 2024-09-16T12:22:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:106450K .......... .......... .......... .......... .......... 63% 26.9M 2s 2024-09-16T12:22:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:106500K .......... .......... .......... .......... .......... 63% 32.9M 2s 2024-09-16T12:22:57.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:106550K .......... .......... .......... .......... .......... 63% 23.3M 2s 2024-09-16T12:22:57.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:106600K .......... .......... .......... .......... .......... 63% 64.6M 2s 2024-09-16T12:22:57.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:106650K .......... .......... .......... .......... .......... 63% 22.5M 2s 2024-09-16T12:22:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:106700K .......... .......... .......... .......... .......... 63% 36.5M 2s 2024-09-16T12:22:57.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:106750K .......... .......... .......... .......... .......... 63% 27.0M 2s 2024-09-16T12:22:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:106800K .......... .......... .......... .......... .......... 63% 25.3M 2s 2024-09-16T12:22:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:106850K .......... .......... .......... .......... .......... 63% 33.1M 2s 2024-09-16T12:22:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:106900K .......... .......... .......... .......... .......... 63% 25.7M 2s 2024-09-16T12:22:57.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:106950K .......... .......... .......... .......... .......... 63% 36.4M 2s 2024-09-16T12:22:57.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:107000K .......... .......... .......... .......... .......... 63% 20.9M 2s 2024-09-16T12:22:57.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:107050K .......... .......... .......... .......... .......... 63% 28.0M 2s 2024-09-16T12:22:57.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:107100K .......... .......... .......... .......... .......... 63% 50.6M 2s 2024-09-16T12:22:57.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:107150K .......... .......... .......... .......... .......... 63% 23.3M 2s 2024-09-16T12:22:57.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:107200K .......... .......... .......... .......... .......... 63% 23.7M 2s 2024-09-16T12:22:57.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:107250K .......... .......... .......... .......... .......... 63% 40.5M 2s 2024-09-16T12:22:57.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:107300K .......... .......... .......... .......... .......... 63% 28.0M 2s 2024-09-16T12:22:57.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:107350K .......... .......... .......... .......... .......... 63% 37.9M 2s 2024-09-16T12:22:57.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:107400K .......... .......... .......... .......... .......... 63% 24.7M 2s 2024-09-16T12:22:57.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:107450K .......... .......... .......... .......... .......... 63% 37.9M 2s 2024-09-16T12:22:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:107500K .......... .......... .......... .......... .......... 63% 19.3M 2s 2024-09-16T12:22:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:107550K .......... .......... .......... .......... .......... 63% 27.1M 2s 2024-09-16T12:22:57.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:107600K .......... .......... .......... .......... .......... 63% 79.0M 2s 2024-09-16T12:22:57.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:107650K .......... .......... .......... .......... .......... 63% 23.6M 2s 2024-09-16T12:22:57.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:107700K .......... .......... .......... .......... .......... 63% 32.6M 2s 2024-09-16T12:22:57.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:107750K .......... .......... .......... .......... .......... 63% 25.3M 2s 2024-09-16T12:22:57.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:107800K .......... .......... .......... .......... .......... 63% 24.3M 2s 2024-09-16T12:22:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:107850K .......... .......... .......... .......... .......... 64% 48.1M 2s 2024-09-16T12:22:57.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:107900K .......... .......... .......... .......... .......... 64% 25.1M 2s 2024-09-16T12:22:57.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:107950K .......... .......... .......... .......... .......... 64% 24.5M 2s 2024-09-16T12:22:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:108000K .......... .......... .......... .......... .......... 64% 30.7M 2s 2024-09-16T12:22:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:108050K .......... .......... .......... .......... .......... 64% 28.2M 2s 2024-09-16T12:22:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:108100K .......... .......... .......... .......... .......... 64% 47.4M 2s 2024-09-16T12:22:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:108150K .......... .......... .......... .......... .......... 64% 23.9M 2s 2024-09-16T12:22:57.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:108200K .......... .......... .......... .......... .......... 64% 36.8M 2s 2024-09-16T12:22:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:108250K .......... .......... .......... .......... .......... 64% 24.0M 2s 2024-09-16T12:22:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:108300K .......... .......... .......... .......... .......... 64% 25.3M 2s 2024-09-16T12:22:57.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:108350K .......... .......... .......... .......... .......... 64% 24.3M 2s 2024-09-16T12:22:57.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:108400K .......... .......... .......... .......... .......... 64% 45.4M 2s 2024-09-16T12:22:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:108450K .......... .......... .......... .......... .......... 64% 24.2M 2s 2024-09-16T12:22:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:108500K .......... .......... .......... .......... .......... 64% 38.0M 2s 2024-09-16T12:22:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:108550K .......... .......... .......... .......... .......... 64% 24.4M 2s 2024-09-16T12:22:57.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:108600K .......... .......... .......... .......... .......... 64% 42.9M 2s 2024-09-16T12:22:57.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:108650K .......... .......... .......... .......... .......... 64% 25.3M 2s 2024-09-16T12:22:57.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:108700K .......... .......... .......... .......... .......... 64% 39.5M 2s 2024-09-16T12:22:57.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:108750K .......... .......... .......... .......... .......... 64% 22.5M 2s 2024-09-16T12:22:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:108800K .......... .......... .......... .......... .......... 64% 22.2M 2s 2024-09-16T12:22:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:108850K .......... .......... .......... .......... .......... 64% 28.2M 2s 2024-09-16T12:22:57.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:108900K .......... .......... .......... .......... .......... 64% 41.0M 2s 2024-09-16T12:22:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:108950K .......... .......... .......... .......... .......... 64% 24.4M 2s 2024-09-16T12:22:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:109000K .......... .......... .......... .......... .......... 64% 36.1M 2s 2024-09-16T12:22:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:109050K .......... .......... .......... .......... .......... 64% 24.8M 2s 2024-09-16T12:22:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:109100K .......... .......... .......... .......... .......... 64% 46.2M 2s 2024-09-16T12:22:57.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:109150K .......... .......... .......... .......... .......... 64% 22.7M 2s 2024-09-16T12:22:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:109200K .......... .......... .......... .......... .......... 64% 24.8M 2s 2024-09-16T12:22:57.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:109250K .......... .......... .......... .......... .......... 64% 40.1M 2s 2024-09-16T12:22:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:109300K .......... .......... .......... .......... .......... 64% 24.8M 2s 2024-09-16T12:22:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:109350K .......... .......... .......... .......... .......... 64% 22.4M 2s 2024-09-16T12:22:57.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:109400K .......... .......... .......... .......... .......... 64% 51.2M 2s 2024-09-16T12:22:57.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:109450K .......... .......... .......... .......... .......... 64% 28.3M 2s 2024-09-16T12:22:57.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:109500K .......... .......... .......... .......... .......... 64% 29.1M 2s 2024-09-16T12:22:57.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:109550K .......... .......... .......... .......... .......... 65% 24.9M 2s 2024-09-16T12:22:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:109600K .......... .......... .......... .......... .......... 65% 24.9M 2s 2024-09-16T12:22:57.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:109650K .......... .......... .......... .......... .......... 65% 43.1M 2s 2024-09-16T12:22:57.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:109700K .......... .......... .......... .......... .......... 65% 25.6M 2s 2024-09-16T12:22:57.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:109750K .......... .......... .......... .......... .......... 65% 36.3M 2s 2024-09-16T12:22:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:109800K .......... .......... .......... .......... .......... 65% 24.1M 2s 2024-09-16T12:22:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:109850K .......... .......... .......... .......... .......... 65% 44.3M 2s 2024-09-16T12:22:57.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:109900K .......... .......... .......... .......... .......... 65% 26.2M 2s 2024-09-16T12:22:57.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:109950K .......... .......... .......... .......... .......... 65% 23.9M 2s 2024-09-16T12:22:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:110000K .......... .......... .......... .......... .......... 65% 33.9M 2s 2024-09-16T12:22:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:110050K .......... .......... .......... .......... .......... 65% 24.7M 2s 2024-09-16T12:22:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:110100K .......... .......... .......... .......... .......... 65% 26.7M 2s 2024-09-16T12:22:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:110150K .......... .......... .......... .......... .......... 65% 40.4M 2s 2024-09-16T12:22:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:110200K .......... .......... .......... .......... .......... 65% 25.0M 2s 2024-09-16T12:22:57.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:110250K .......... .......... .......... .......... .......... 65% 45.9M 2s 2024-09-16T12:22:57.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:110300K .......... .......... .......... .......... .......... 65% 21.5M 2s 2024-09-16T12:22:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:110350K .......... .......... .......... .......... .......... 65% 25.5M 2s 2024-09-16T12:22:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:110400K .......... .......... .......... .......... .......... 65% 44.7M 2s 2024-09-16T12:22:57.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:110450K .......... .......... .......... .......... .......... 65% 25.6M 2s 2024-09-16T12:22:57.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:110500K .......... .......... .......... .......... .......... 65% 33.4M 2s 2024-09-16T12:22:57.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:110550K .......... .......... .......... .......... .......... 65% 23.8M 2s 2024-09-16T12:22:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:110600K .......... .......... .......... .......... .......... 65% 24.3M 2s 2024-09-16T12:22:57.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:110650K .......... .......... .......... .......... .......... 65% 44.0M 2s 2024-09-16T12:22:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:110700K .......... .......... .......... .......... .......... 65% 24.6M 2s 2024-09-16T12:22:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:110750K .......... .......... .......... .......... .......... 65% 25.4M 2s 2024-09-16T12:22:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:110800K .......... .......... .......... .......... .......... 65% 33.3M 2s 2024-09-16T12:22:57.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:110850K .......... .......... .......... .......... .......... 65% 33.2M 2s 2024-09-16T12:22:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:110900K .......... .......... .......... .......... .......... 65% 33.9M 2s 2024-09-16T12:22:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:110950K .......... .......... .......... .......... .......... 65% 25.7M 2s 2024-09-16T12:22:57.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:111000K .......... .......... .......... .......... .......... 65% 34.5M 2s 2024-09-16T12:22:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:111050K .......... .......... .......... .......... .......... 65% 24.0M 2s 2024-09-16T12:22:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:111100K .......... .......... .......... .......... .......... 65% 24.6M 2s 2024-09-16T12:22:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:111150K .......... .......... .......... .......... .......... 65% 24.6M 2s 2024-09-16T12:22:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:111200K .......... .......... .......... .......... .......... 65% 52.0M 2s 2024-09-16T12:22:57.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:111250K .......... .......... .......... .......... .......... 66% 22.0M 2s 2024-09-16T12:22:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:111300K .......... .......... .......... .......... .......... 66% 36.7M 2s 2024-09-16T12:22:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:111350K .......... .......... .......... .......... .......... 66% 27.1M 2s 2024-09-16T12:22:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:111400K .......... .......... .......... .......... .......... 66% 50.4M 2s 2024-09-16T12:22:57.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:111450K .......... .......... .......... .......... .......... 66% 24.2M 2s 2024-09-16T12:22:57.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:111500K .......... .......... .......... .......... .......... 66% 34.7M 2s 2024-09-16T12:22:57.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:111550K .......... .......... .......... .......... .......... 66% 24.3M 2s 2024-09-16T12:22:57.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:111600K .......... .......... .......... .......... .......... 66% 22.5M 2s 2024-09-16T12:22:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:111650K .......... .......... .......... .......... .......... 66% 26.6M 2s 2024-09-16T12:22:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:111700K .......... .......... .......... .......... .......... 66% 53.9M 2s 2024-09-16T12:22:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:111750K .......... .......... .......... .......... .......... 66% 21.6M 2s 2024-09-16T12:22:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:111800K .......... .......... .......... .......... .......... 66% 39.3M 2s 2024-09-16T12:22:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:111850K .......... .......... .......... .......... .......... 66% 25.0M 2s 2024-09-16T12:22:57.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:111900K .......... .......... .......... .......... .......... 66% 48.5M 2s 2024-09-16T12:22:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:111950K .......... .......... .......... .......... .......... 66% 24.3M 2s 2024-09-16T12:22:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:112000K .......... .......... .......... .......... .......... 66% 23.5M 2s 2024-09-16T12:22:57.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:112050K .......... .......... .......... .......... .......... 66% 41.5M 2s 2024-09-16T12:22:57.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:112100K .......... .......... .......... .......... .......... 66% 24.5M 2s 2024-09-16T12:22:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:112150K .......... .......... .......... .......... .......... 66% 21.9M 2s 2024-09-16T12:22:57.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:112200K .......... .......... .......... .......... .......... 66% 56.8M 2s 2024-09-16T12:22:57.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:112250K .......... .......... .......... .......... .......... 66% 33.3M 2s 2024-09-16T12:22:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:112300K .......... .......... .......... .......... .......... 66% 24.6M 2s 2024-09-16T12:22:57.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:112350K .......... .......... .......... .......... .......... 66% 24.4M 2s 2024-09-16T12:22:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:112400K .......... .......... .......... .......... .......... 66% 25.0M 2s 2024-09-16T12:22:57.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:112450K .......... .......... .......... .......... .......... 66% 49.3M 2s 2024-09-16T12:22:57.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:112500K .......... .......... .......... .......... .......... 66% 24.9M 2s 2024-09-16T12:22:57.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:112550K .......... .......... .......... .......... .......... 66% 35.3M 2s 2024-09-16T12:22:57.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:112600K .......... .......... .......... .......... .......... 66% 24.1M 2s 2024-09-16T12:22:57.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:112650K .......... .......... .......... .......... .......... 66% 47.7M 2s 2024-09-16T12:22:57.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:112700K .......... .......... .......... .......... .......... 66% 25.0M 2s 2024-09-16T12:22:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:112750K .......... .......... .......... .......... .......... 66% 24.8M 2s 2024-09-16T12:22:57.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:112800K .......... .......... .......... .......... .......... 66% 33.4M 2s 2024-09-16T12:22:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:112850K .......... .......... .......... .......... .......... 66% 22.9M 2s 2024-09-16T12:22:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:112900K .......... .......... .......... .......... .......... 67% 22.0M 2s 2024-09-16T12:22:57.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:112950K .......... .......... .......... .......... .......... 67% 75.9M 2s 2024-09-16T12:22:57.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:113000K .......... .......... .......... .......... .......... 67% 23.1M 2s 2024-09-16T12:22:57.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:113050K .......... .......... .......... .......... .......... 67% 36.7M 2s 2024-09-16T12:22:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:113100K .......... .......... .......... .......... .......... 67% 24.8M 2s 2024-09-16T12:22:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:113150K .......... .......... .......... .......... .......... 67% 25.0M 2s 2024-09-16T12:22:57.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:113200K .......... .......... .......... .......... .......... 67% 43.8M 2s 2024-09-16T12:22:57.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:113250K .......... .......... .......... .......... .......... 67% 26.0M 2s 2024-09-16T12:22:57.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:113300K .......... .......... .......... .......... .......... 67% 34.8M 2s 2024-09-16T12:22:57.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:113350K .......... .......... .......... .......... .......... 67% 22.6M 2s 2024-09-16T12:22:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:113400K .......... .......... .......... .......... .......... 67% 25.6M 2s 2024-09-16T12:22:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:113450K .......... .......... .......... .......... .......... 67% 55.5M 2s 2024-09-16T12:22:57.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:113500K .......... .......... .......... .......... .......... 67% 23.8M 2s 2024-09-16T12:22:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:113550K .......... .......... .......... .......... .......... 67% 23.7M 2s 2024-09-16T12:22:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:113600K .......... .......... .......... .......... .......... 67% 35.3M 2s 2024-09-16T12:22:57.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:113650K .......... .......... .......... .......... .......... 67% 25.1M 2s 2024-09-16T12:22:57.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:113700K .......... .......... .......... .......... .......... 67% 48.1M 2s 2024-09-16T12:22:57.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:113750K .......... .......... .......... .......... .......... 67% 24.3M 2s 2024-09-16T12:22:57.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:113800K .......... .......... .......... .......... .......... 67% 36.9M 2s 2024-09-16T12:22:57.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:113850K .......... .......... .......... .......... .......... 67% 25.6M 2s 2024-09-16T12:22:57.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:113900K .......... .......... .......... .......... .......... 67% 22.7M 2s 2024-09-16T12:22:57.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:113950K .......... .......... .......... .......... .......... 67% 24.9M 2s 2024-09-16T12:22:57.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:114000K .......... .......... .......... .......... .......... 67% 46.3M 2s 2024-09-16T12:22:57.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:114050K .......... .......... .......... .......... .......... 67% 25.1M 2s 2024-09-16T12:22:57.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:114100K .......... .......... .......... .......... .......... 67% 38.0M 2s 2024-09-16T12:22:57.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:114150K .......... .......... .......... .......... .......... 67% 23.5M 2s 2024-09-16T12:22:57.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:114200K .......... .......... .......... .......... .......... 67% 45.9M 2s 2024-09-16T12:22:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:114250K .......... .......... .......... .......... .......... 67% 25.1M 2s 2024-09-16T12:22:57.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:114300K .......... .......... .......... .......... .......... 67% 35.4M 2s 2024-09-16T12:22:57.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:114350K .......... .......... .......... .......... .......... 67% 19.5M 2s 2024-09-16T12:22:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:114400K .......... .......... .......... .......... .......... 67% 20.5M 2s 2024-09-16T12:22:57.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:114450K .......... .......... .......... .......... .......... 67% 23.9M 2s 2024-09-16T12:22:57.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:114500K .......... .......... .......... .......... .......... 67% 51.1M 2s 2024-09-16T12:22:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:114550K .......... .......... .......... .......... .......... 67% 32.5M 2s 2024-09-16T12:22:57.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:114600K .......... .......... .......... .......... .......... 68% 21.9M 2s 2024-09-16T12:22:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:114650K .......... .......... .......... .......... .......... 68% 29.0M 2s 2024-09-16T12:22:57.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:114700K .......... .......... .......... .......... .......... 68% 72.0M 2s 2024-09-16T12:22:57.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:114750K .......... .......... .......... .......... .......... 68% 20.1M 2s 2024-09-16T12:22:57.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:114800K .......... .......... .......... .......... .......... 68% 25.6M 2s 2024-09-16T12:22:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:114850K .......... .......... .......... .......... .......... 68% 36.6M 2s 2024-09-16T12:22:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:114900K .......... .......... .......... .......... .......... 68% 25.9M 2s 2024-09-16T12:22:57.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:114950K .......... .......... .......... .......... .......... 68% 46.7M 2s 2024-09-16T12:22:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:115000K .......... .......... .......... .......... .......... 68% 24.5M 2s 2024-09-16T12:22:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:115050K .......... .......... .......... .......... .......... 68% 30.7M 2s 2024-09-16T12:22:57.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:115100K .......... .......... .......... .......... .......... 68% 24.7M 2s 2024-09-16T12:22:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:115150K .......... .......... .......... .......... .......... 68% 23.7M 2s 2024-09-16T12:22:57.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:115200K .......... .......... .......... .......... .......... 68% 25.0M 2s 2024-09-16T12:22:57.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:115250K .......... .......... .......... .......... .......... 68% 50.7M 2s 2024-09-16T12:22:57.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:115300K .......... .......... .......... .......... .......... 68% 25.0M 2s 2024-09-16T12:22:57.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:115350K .......... .......... .......... .......... .......... 68% 34.4M 2s 2024-09-16T12:22:57.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:115400K .......... .......... .......... .......... .......... 68% 24.5M 2s 2024-09-16T12:22:57.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:115450K .......... .......... .......... .......... .......... 68% 47.6M 2s 2024-09-16T12:22:57.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:115500K .......... .......... .......... .......... .......... 68% 24.6M 2s 2024-09-16T12:22:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:115550K .......... .......... .......... .......... .......... 68% 24.6M 2s 2024-09-16T12:22:57.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:115600K .......... .......... .......... .......... .......... 68% 34.2M 2s 2024-09-16T12:22:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:115650K .......... .......... .......... .......... .......... 68% 24.5M 2s 2024-09-16T12:22:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:115700K .......... .......... .......... .......... .......... 68% 24.4M 2s 2024-09-16T12:22:57.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:115750K .......... .......... .......... .......... .......... 68% 63.4M 2s 2024-09-16T12:22:57.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:115800K .......... .......... .......... .......... .......... 68% 21.4M 2s 2024-09-16T12:22:57.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:115850K .......... .......... .......... .......... .......... 68% 38.5M 2s 2024-09-16T12:22:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:115900K .......... .......... .......... .......... .......... 68% 24.4M 2s 2024-09-16T12:22:57.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:115950K .......... .......... .......... .......... .......... 68% 24.0M 2s 2024-09-16T12:22:57.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:116000K .......... .......... .......... .......... .......... 68% 48.4M 2s 2024-09-16T12:22:57.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:116050K .......... .......... .......... .......... .......... 68% 24.3M 2s 2024-09-16T12:22:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:116100K .......... .......... .......... .......... .......... 68% 36.1M 2s 2024-09-16T12:22:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:116150K .......... .......... .......... .......... .......... 68% 22.8M 2s 2024-09-16T12:22:57.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:116200K .......... .......... .......... .......... .......... 68% 25.5M 2s 2024-09-16T12:22:57.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:116250K .......... .......... .......... .......... .......... 68% 46.8M 2s 2024-09-16T12:22:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:116300K .......... .......... .......... .......... .......... 69% 23.8M 2s 2024-09-16T12:22:57.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:116350K .......... .......... .......... .......... .......... 69% 24.3M 2s 2024-09-16T12:22:57.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:116400K .......... .......... .......... .......... .......... 69% 36.4M 2s 2024-09-16T12:22:57.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:116450K .......... .......... .......... .......... .......... 69% 25.2M 2s 2024-09-16T12:22:57.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:116500K .......... .......... .......... .......... .......... 69% 45.1M 2s 2024-09-16T12:22:57.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:116550K .......... .......... .......... .......... .......... 69% 25.0M 2s 2024-09-16T12:22:57.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:116600K .......... .......... .......... .......... .......... 69% 36.9M 2s 2024-09-16T12:22:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:116650K .......... .......... .......... .......... .......... 69% 25.2M 2s 2024-09-16T12:22:57.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:116700K .......... .......... .......... .......... .......... 69% 22.8M 2s 2024-09-16T12:22:57.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:116750K .......... .......... .......... .......... .......... 69% 25.3M 2s 2024-09-16T12:22:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:116800K .......... .......... .......... .......... .......... 69% 41.8M 2s 2024-09-16T12:22:57.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:116850K .......... .......... .......... .......... .......... 69% 25.7M 2s 2024-09-16T12:22:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:116900K .......... .......... .......... .......... .......... 69% 39.0M 2s 2024-09-16T12:22:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:116950K .......... .......... .......... .......... .......... 69% 31.7M 2s 2024-09-16T12:22:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:117000K .......... .......... .......... .......... .......... 69% 37.1M 2s 2024-09-16T12:22:57.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:117050K .......... .......... .......... .......... .......... 69% 22.7M 2s 2024-09-16T12:22:57.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:117100K .......... .......... .......... .......... .......... 69% 38.5M 2s 2024-09-16T12:22:57.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:117150K .......... .......... .......... .......... .......... 69% 22.9M 2s 2024-09-16T12:22:57.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:117200K .......... .......... .......... .......... .......... 69% 30.2M 2s 2024-09-16T12:22:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:117250K .......... .......... .......... .......... .......... 69% 19.8M 2s 2024-09-16T12:22:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:117300K .......... .......... .......... .......... .......... 69% 56.4M 2s 2024-09-16T12:22:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:117350K .......... .......... .......... .......... .......... 69% 24.0M 2s 2024-09-16T12:22:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:117400K .......... .......... .......... .......... .......... 69% 32.7M 2s 2024-09-16T12:22:57.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:117450K .......... .......... .......... .......... .......... 69% 25.4M 2s 2024-09-16T12:22:57.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:117500K .......... .......... .......... .......... .......... 69% 48.9M 2s 2024-09-16T12:22:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:117550K .......... .......... .......... .......... .......... 69% 23.7M 2s 2024-09-16T12:22:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:117600K .......... .......... .......... .......... .......... 69% 25.0M 2s 2024-09-16T12:22:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:117650K .......... .......... .......... .......... .......... 69% 39.4M 2s 2024-09-16T12:22:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:117700K .......... .......... .......... .......... .......... 69% 22.9M 2s 2024-09-16T12:22:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:117750K .......... .......... .......... .......... .......... 69% 42.4M 2s 2024-09-16T12:22:57.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:117800K .......... .......... .......... .......... .......... 69% 24.6M 2s 2024-09-16T12:22:57.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:117850K .......... .......... .......... .......... .......... 69% 35.8M 2s 2024-09-16T12:22:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:117900K .......... .......... .......... .......... .......... 69% 24.7M 2s 2024-09-16T12:22:57.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:117950K .......... .......... .......... .......... .......... 70% 23.8M 2s 2024-09-16T12:22:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:118000K .......... .......... .......... .......... .......... 70% 26.9M 2s 2024-09-16T12:22:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:118050K .......... .......... .......... .......... .......... 70% 39.9M 2s 2024-09-16T12:22:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:118100K .......... .......... .......... .......... .......... 70% 25.6M 2s 2024-09-16T12:22:57.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:118150K .......... .......... .......... .......... .......... 70% 19.0M 2s 2024-09-16T12:22:57.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:118200K .......... .......... .......... .......... .......... 70% 36.5M 2s 2024-09-16T12:22:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:118250K .......... .......... .......... .......... .......... 70% 34.3M 2s 2024-09-16T12:22:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:118300K .......... .......... .......... .......... .......... 70% 24.8M 2s 2024-09-16T12:22:57.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:118350K .......... .......... .......... .......... .......... 70% 23.3M 2s 2024-09-16T12:22:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:118400K .......... .......... .......... .......... .......... 70% 34.6M 2s 2024-09-16T12:22:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:118450K .......... .......... .......... .......... .......... 70% 24.8M 2s 2024-09-16T12:22:57.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:118500K .......... .......... .......... .......... .......... 70% 25.5M 2s 2024-09-16T12:22:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:118550K .......... .......... .......... .......... .......... 70% 45.2M 2s 2024-09-16T12:22:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:118600K .......... .......... .......... .......... .......... 70% 23.4M 2s 2024-09-16T12:22:57.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:118650K .......... .......... .......... .......... .......... 70% 40.2M 2s 2024-09-16T12:22:57.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:118700K .......... .......... .......... .......... .......... 70% 24.9M 2s 2024-09-16T12:22:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:118750K .......... .......... .......... .......... .......... 70% 23.7M 2s 2024-09-16T12:22:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:118800K .......... .......... .......... .......... .......... 70% 44.3M 2s 2024-09-16T12:22:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:118850K .......... .......... .......... .......... .......... 70% 25.4M 2s 2024-09-16T12:22:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:118900K .......... .......... .......... .......... .......... 70% 38.5M 2s 2024-09-16T12:22:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:118950K .......... .......... .......... .......... .......... 70% 38.8M 2s 2024-09-16T12:22:57.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:119000K .......... .......... .......... .......... .......... 70% 22.9M 2s 2024-09-16T12:22:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:119050K .......... .......... .......... .......... .......... 70% 33.0M 2s 2024-09-16T12:22:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:119100K .......... .......... .......... .......... .......... 70% 21.4M 2s 2024-09-16T12:22:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:119150K .......... .......... .......... .......... .......... 70% 25.7M 2s 2024-09-16T12:22:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:119200K .......... .......... .......... .......... .......... 70% 34.7M 2s 2024-09-16T12:22:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:119250K .......... .......... .......... .......... .......... 70% 26.0M 2s 2024-09-16T12:22:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:119300K .......... .......... .......... .......... .......... 70% 55.5M 2s 2024-09-16T12:22:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:119350K .......... .......... .......... .......... .......... 70% 21.1M 2s 2024-09-16T12:22:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:119400K .......... .......... .......... .......... .......... 70% 38.5M 2s 2024-09-16T12:22:57.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:119450K .......... .......... .......... .......... .......... 70% 25.0M 2s 2024-09-16T12:22:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:119500K .......... .......... .......... .......... .......... 70% 47.7M 2s 2024-09-16T12:22:57.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:119550K .......... .......... .......... .......... .......... 70% 19.0M 2s 2024-09-16T12:22:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:119600K .......... .......... .......... .......... .......... 70% 40.4M 2s 2024-09-16T12:22:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:119650K .......... .......... .......... .......... .......... 71% 20.8M 2s 2024-09-16T12:22:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:119700K .......... .......... .......... .......... .......... 71% 39.3M 2s 2024-09-16T12:22:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:119750K .......... .......... .......... .......... .......... 71% 23.3M 2s 2024-09-16T12:22:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:119800K .......... .......... .......... .......... .......... 71% 43.5M 2s 2024-09-16T12:22:57.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:119850K .......... .......... .......... .......... .......... 71% 24.6M 2s 2024-09-16T12:22:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:119900K .......... .......... .......... .......... .......... 71% 39.7M 2s 2024-09-16T12:22:57.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:119950K .......... .......... .......... .......... .......... 71% 23.3M 2s 2024-09-16T12:22:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:120000K .......... .......... .......... .......... .......... 71% 23.7M 2s 2024-09-16T12:22:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:120050K .......... .......... .......... .......... .......... 71% 47.9M 2s 2024-09-16T12:22:57.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:120100K .......... .......... .......... .......... .......... 71% 25.8M 2s 2024-09-16T12:22:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:120150K .......... .......... .......... .......... .......... 71% 31.9M 2s 2024-09-16T12:22:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:120200K .......... .......... .......... .......... .......... 71% 25.0M 2s 2024-09-16T12:22:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:120250K .......... .......... .......... .......... .......... 71% 25.0M 2s 2024-09-16T12:22:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:120300K .......... .......... .......... .......... .......... 71% 43.2M 2s 2024-09-16T12:22:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:120350K .......... .......... .......... .......... .......... 71% 25.5M 2s 2024-09-16T12:22:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:120400K .......... .......... .......... .......... .......... 71% 23.6M 2s 2024-09-16T12:22:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:120450K .......... .......... .......... .......... .......... 71% 38.9M 2s 2024-09-16T12:22:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:120500K .......... .......... .......... .......... .......... 71% 25.4M 2s 2024-09-16T12:22:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:120550K .......... .......... .......... .......... .......... 71% 42.1M 2s 2024-09-16T12:22:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:120600K .......... .......... .......... .......... .......... 71% 24.8M 2s 2024-09-16T12:22:57.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:120650K .......... .......... .......... .......... .......... 71% 41.0M 2s 2024-09-16T12:22:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:120700K .......... .......... .......... .......... .......... 71% 22.4M 2s 2024-09-16T12:22:57.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:120750K .......... .......... .......... .......... .......... 71% 23.5M 2s 2024-09-16T12:22:57.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:120800K .......... .......... .......... .......... .......... 71% 26.1M 2s 2024-09-16T12:22:57.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:120850K .......... .......... .......... .......... .......... 71% 41.6M 2s 2024-09-16T12:22:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:120900K .......... .......... .......... .......... .......... 71% 34.3M 2s 2024-09-16T12:22:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:120950K .......... .......... .......... .......... .......... 71% 25.9M 2s 2024-09-16T12:22:57.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:121000K .......... .......... .......... .......... .......... 71% 26.1M 2s 2024-09-16T12:22:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:121050K .......... .......... .......... .......... .......... 71% 43.1M 2s 2024-09-16T12:22:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:121100K .......... .......... .......... .......... .......... 71% 25.2M 2s 2024-09-16T12:22:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:121150K .......... .......... .......... .......... .......... 71% 24.4M 2s 2024-09-16T12:22:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:121200K .......... .......... .......... .......... .......... 71% 36.9M 2s 2024-09-16T12:22:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:121250K .......... .......... .......... .......... .......... 71% 23.6M 2s 2024-09-16T12:22:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:121300K .......... .......... .......... .......... .......... 71% 25.5M 2s 2024-09-16T12:22:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:121350K .......... .......... .......... .......... .......... 72% 44.3M 2s 2024-09-16T12:22:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:121400K .......... .......... .......... .......... .......... 72% 23.8M 2s 2024-09-16T12:22:57.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:121450K .......... .......... .......... .......... .......... 72% 38.0M 2s 2024-09-16T12:22:57.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:121500K .......... .......... .......... .......... .......... 72% 25.5M 2s 2024-09-16T12:22:57.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:121550K .......... .......... .......... .......... .......... 72% 23.8M 2s 2024-09-16T12:22:57.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:121600K .......... .......... .......... .......... .......... 72% 43.9M 2s 2024-09-16T12:22:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:121650K .......... .......... .......... .......... .......... 72% 25.5M 2s 2024-09-16T12:22:57.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:121700K .......... .......... .......... .......... .......... 72% 38.4M 2s 2024-09-16T12:22:57.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:121750K .......... .......... .......... .......... .......... 72% 24.9M 2s 2024-09-16T12:22:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:121800K .......... .......... .......... .......... .......... 72% 22.7M 2s 2024-09-16T12:22:57.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:121850K .......... .......... .......... .......... .......... 72% 53.4M 2s 2024-09-16T12:22:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:121900K .......... .......... .......... .......... .......... 72% 23.5M 2s 2024-09-16T12:22:57.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:121950K .......... .......... .......... .......... .......... 72% 18.0M 2s 2024-09-16T12:22:57.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:122000K .......... .......... .......... .......... .......... 72% 35.8M 2s 2024-09-16T12:22:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:122050K .......... .......... .......... .......... .......... 72% 25.5M 2s 2024-09-16T12:22:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:122100K .......... .......... .......... .......... .......... 72% 46.8M 2s 2024-09-16T12:22:57.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:122150K .......... .......... .......... .......... .......... 72% 23.7M 2s 2024-09-16T12:22:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:122200K .......... .......... .......... .......... .......... 72% 37.7M 2s 2024-09-16T12:22:57.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:122250K .......... .......... .......... .......... .......... 72% 23.7M 2s 2024-09-16T12:22:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:122300K .......... .......... .......... .......... .......... 72% 48.1M 2s 2024-09-16T12:22:57.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:122350K .......... .......... .......... .......... .......... 72% 16.1M 2s 2024-09-16T12:22:57.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:122400K .......... .......... .......... .......... .......... 72% 54.8M 2s 2024-09-16T12:22:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:122450K .......... .......... .......... .......... .......... 72% 33.3M 2s 2024-09-16T12:22:57.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:122500K .......... .......... .......... .......... .......... 72% 24.7M 2s 2024-09-16T12:22:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:122550K .......... .......... .......... .......... .......... 72% 24.3M 2s 2024-09-16T12:22:57.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:122600K .......... .......... .......... .......... .......... 72% 47.8M 2s 2024-09-16T12:22:57.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:122650K .......... .......... .......... .......... .......... 72% 25.1M 2s 2024-09-16T12:22:57.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:122700K .......... .......... .......... .......... .......... 72% 36.5M 2s 2024-09-16T12:22:57.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:122750K .......... .......... .......... .......... .......... 72% 24.1M 2s 2024-09-16T12:22:57.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:122800K .......... .......... .......... .......... .......... 72% 24.2M 2s 2024-09-16T12:22:57.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:122850K .......... .......... .......... .......... .......... 72% 45.6M 2s 2024-09-16T12:22:57.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:122900K .......... .......... .......... .......... .......... 72% 25.0M 2s 2024-09-16T12:22:57.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:122950K .......... .......... .......... .......... .......... 72% 33.9M 2s 2024-09-16T12:22:57.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:123000K .......... .......... .......... .......... .......... 72% 24.3M 2s 2024-09-16T12:22:57.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:123050K .......... .......... .......... .......... .......... 73% 26.3M 2s 2024-09-16T12:22:57.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:123100K .......... .......... .......... .......... .......... 73% 44.1M 2s 2024-09-16T12:22:57.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:123150K .......... .......... .......... .......... .......... 73% 23.2M 2s 2024-09-16T12:22:57.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:123200K .......... .......... .......... .......... .......... 73% 25.0M 2s 2024-09-16T12:22:57.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:123250K .......... .......... .......... .......... .......... 73% 36.5M 2s 2024-09-16T12:22:57.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:123300K .......... .......... .......... .......... .......... 73% 25.6M 2s 2024-09-16T12:22:57.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:123350K .......... .......... .......... .......... .......... 73% 41.3M 2s 2024-09-16T12:22:57.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:123400K .......... .......... .......... .......... .......... 73% 25.4M 2s 2024-09-16T12:22:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:123450K .......... .......... .......... .......... .......... 73% 37.0M 2s 2024-09-16T12:22:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:123500K .......... .......... .......... .......... .......... 73% 23.6M 2s 2024-09-16T12:22:57.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:123550K .......... .......... .......... .......... .......... 73% 24.8M 2s 2024-09-16T12:22:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:123600K .......... .......... .......... .......... .......... 73% 24.5M 2s 2024-09-16T12:22:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:123650K .......... .......... .......... .......... .......... 73% 48.5M 2s 2024-09-16T12:22:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:123700K .......... .......... .......... .......... .......... 73% 23.8M 2s 2024-09-16T12:22:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:123750K .......... .......... .......... .......... .......... 73% 39.2M 2s 2024-09-16T12:22:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:123800K .......... .......... .......... .......... .......... 73% 24.0M 2s 2024-09-16T12:22:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:123850K .......... .......... .......... .......... .......... 73% 52.9M 2s 2024-09-16T12:22:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:123900K .......... .......... .......... .......... .......... 73% 33.8M 2s 2024-09-16T12:22:57.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:123950K .......... .......... .......... .......... .......... 73% 18.6M 2s 2024-09-16T12:22:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:124000K .......... .......... .......... .......... .......... 73% 36.8M 2s 2024-09-16T12:22:57.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:124050K .......... .......... .......... .......... .......... 73% 23.7M 2s 2024-09-16T12:22:57.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:124100K .......... .......... .......... .......... .......... 73% 25.6M 2s 2024-09-16T12:22:57.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:124150K .......... .......... .......... .......... .......... 73% 40.5M 2s 2024-09-16T12:22:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:124200K .......... .......... .......... .......... .......... 73% 25.1M 2s 2024-09-16T12:22:57.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:124250K .......... .......... .......... .......... .......... 73% 30.5M 2s 2024-09-16T12:22:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:124300K .......... .......... .......... .......... .......... 73% 16.7M 2s 2024-09-16T12:22:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:124350K .......... .......... .......... .......... .......... 73% 72.6M 2s 2024-09-16T12:22:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:124400K .......... .......... .......... .......... .......... 73% 54.7M 2s 2024-09-16T12:22:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:124450K .......... .......... .......... .......... .......... 73% 22.5M 2s 2024-09-16T12:22:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:124500K .......... .......... .......... .......... .......... 73% 38.4M 2s 2024-09-16T12:22:57.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:124550K .......... .......... .......... .......... .......... 73% 23.3M 2s 2024-09-16T12:22:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:124600K .......... .......... .......... .......... .......... 73% 23.8M 2s 2024-09-16T12:22:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:124650K .......... .......... .......... .......... .......... 73% 52.2M 2s 2024-09-16T12:22:57.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:124700K .......... .......... .......... .......... .......... 74% 35.2M 2s 2024-09-16T12:22:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:124750K .......... .......... .......... .......... .......... 74% 17.9M 2s 2024-09-16T12:22:57.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:124800K .......... .......... .......... .......... .......... 74% 36.9M 2s 2024-09-16T12:22:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:124850K .......... .......... .......... .......... .......... 74% 25.8M 2s 2024-09-16T12:22:57.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:124900K .......... .......... .......... .......... .......... 74% 45.2M 2s 2024-09-16T12:22:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:124950K .......... .......... .......... .......... .......... 74% 24.0M 1s 2024-09-16T12:22:57.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:125000K .......... .......... .......... .......... .......... 74% 39.2M 1s 2024-09-16T12:22:57.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:125050K .......... .......... .......... .......... .......... 74% 25.6M 1s 2024-09-16T12:22:57.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:125100K .......... .......... .......... .......... .......... 74% 47.5M 1s 2024-09-16T12:22:57.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:125150K .......... .......... .......... .......... .......... 74% 22.2M 1s 2024-09-16T12:22:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:125200K .......... .......... .......... .......... .......... 74% 26.8M 1s 2024-09-16T12:22:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:125250K .......... .......... .......... .......... .......... 74% 32.0M 1s 2024-09-16T12:22:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:125300K .......... .......... .......... .......... .......... 74% 24.4M 1s 2024-09-16T12:22:57.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:125350K .......... .......... .......... .......... .......... 74% 24.7M 1s 2024-09-16T12:22:57.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:125400K .......... .......... .......... .......... .......... 74% 47.3M 1s 2024-09-16T12:22:57.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:125450K .......... .......... .......... .......... .......... 74% 24.5M 1s 2024-09-16T12:22:57.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:125500K .......... .......... .......... .......... .......... 74% 37.4M 1s 2024-09-16T12:22:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:125550K .......... .......... .......... .......... .......... 74% 23.8M 1s 2024-09-16T12:22:58.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:125600K .......... .......... .......... .......... .......... 74% 26.5M 1s 2024-09-16T12:22:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:125650K .......... .......... .......... .......... .......... 74% 43.1M 1s 2024-09-16T12:22:58.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:125700K .......... .......... .......... .......... .......... 74% 24.4M 1s 2024-09-16T12:22:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:125750K .......... .......... .......... .......... .......... 74% 37.1M 1s 2024-09-16T12:22:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:125800K .......... .......... .......... .......... .......... 74% 24.8M 1s 2024-09-16T12:22:58.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:125850K .......... .......... .......... .......... .......... 74% 22.5M 1s 2024-09-16T12:22:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:125900K .......... .......... .......... .......... .......... 74% 45.8M 1s 2024-09-16T12:22:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:125950K .......... .......... .......... .......... .......... 74% 22.4M 1s 2024-09-16T12:22:58.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:126000K .......... .......... .......... .......... .......... 74% 25.6M 1s 2024-09-16T12:22:58.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:126050K .......... .......... .......... .......... .......... 74% 35.1M 1s 2024-09-16T12:22:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:126100K .......... .......... .......... .......... .......... 74% 25.6M 1s 2024-09-16T12:22:58.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:126150K .......... .......... .......... .......... .......... 74% 46.5M 1s 2024-09-16T12:22:58.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:126200K .......... .......... .......... .......... .......... 74% 25.1M 1s 2024-09-16T12:22:58.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:126250K .......... .......... .......... .......... .......... 74% 41.6M 1s 2024-09-16T12:22:58.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:126300K .......... .......... .......... .......... .......... 74% 22.9M 1s 2024-09-16T12:22:58.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:126350K .......... .......... .......... .......... .......... 74% 23.8M 1s 2024-09-16T12:22:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:126400K .......... .......... .......... .......... .......... 75% 24.0M 1s 2024-09-16T12:22:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:126450K .......... .......... .......... .......... .......... 75% 52.8M 1s 2024-09-16T12:22:58.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:126500K .......... .......... .......... .......... .......... 75% 23.3M 1s 2024-09-16T12:22:58.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:126550K .......... .......... .......... .......... .......... 75% 35.5M 1s 2024-09-16T12:22:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:126600K .......... .......... .......... .......... .......... 75% 24.6M 1s 2024-09-16T12:22:58.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:126650K .......... .......... .......... .......... .......... 75% 55.3M 1s 2024-09-16T12:22:58.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:126700K .......... .......... .......... .......... .......... 75% 28.9M 1s 2024-09-16T12:22:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:126750K .......... .......... .......... .......... .......... 75% 20.1M 1s 2024-09-16T12:22:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:126800K .......... .......... .......... .......... .......... 75% 34.9M 1s 2024-09-16T12:22:58.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:126850K .......... .......... .......... .......... .......... 75% 25.9M 1s 2024-09-16T12:22:58.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:126900K .......... .......... .......... .......... .......... 75% 23.3M 1s 2024-09-16T12:22:58.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:126950K .......... .......... .......... .......... .......... 75% 48.3M 1s 2024-09-16T12:22:58.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:127000K .......... .......... .......... .......... .......... 75% 35.2M 1s 2024-09-16T12:22:58.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:127050K .......... .......... .......... .......... .......... 75% 24.4M 1s 2024-09-16T12:22:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:127100K .......... .......... .......... .......... .......... 75% 24.6M 1s 2024-09-16T12:22:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:127150K .......... .......... .......... .......... .......... 75% 23.0M 1s 2024-09-16T12:22:58.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:127200K .......... .......... .......... .......... .......... 75% 46.2M 1s 2024-09-16T12:22:58.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:127250K .......... .......... .......... .......... .......... 75% 25.3M 1s 2024-09-16T12:22:58.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:127300K .......... .......... .......... .......... .......... 75% 36.8M 1s 2024-09-16T12:22:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:127350K .......... .......... .......... .......... .......... 75% 24.0M 1s 2024-09-16T12:22:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:127400K .......... .......... .......... .......... .......... 75% 48.5M 1s 2024-09-16T12:22:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:127450K .......... .......... .......... .......... .......... 75% 25.3M 1s 2024-09-16T12:22:58.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:127500K .......... .......... .......... .......... .......... 75% 32.6M 1s 2024-09-16T12:22:58.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:127550K .......... .......... .......... .......... .......... 75% 23.7M 1s 2024-09-16T12:22:58.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:127600K .......... .......... .......... .......... .......... 75% 26.4M 1s 2024-09-16T12:22:58.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:127650K .......... .......... .......... .......... .......... 75% 24.8M 1s 2024-09-16T12:22:58.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:127700K .......... .......... .......... .......... .......... 75% 47.7M 1s 2024-09-16T12:22:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:127750K .......... .......... .......... .......... .......... 75% 23.5M 1s 2024-09-16T12:22:58.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:127800K .......... .......... .......... .......... .......... 75% 37.2M 1s 2024-09-16T12:22:58.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:127850K .......... .......... .......... .......... .......... 75% 25.2M 1s 2024-09-16T12:22:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:127900K .......... .......... .......... .......... .......... 75% 43.9M 1s 2024-09-16T12:22:58.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:127950K .......... .......... .......... .......... .......... 75% 24.8M 1s 2024-09-16T12:22:58.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:128000K .......... .......... .......... .......... .......... 75% 24.5M 1s 2024-09-16T12:22:58.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:128050K .......... .......... .......... .......... .......... 75% 33.9M 1s 2024-09-16T12:22:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:128100K .......... .......... .......... .......... .......... 76% 24.4M 1s 2024-09-16T12:22:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:128150K .......... .......... .......... .......... .......... 76% 25.2M 1s 2024-09-16T12:22:58.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:128200K .......... .......... .......... .......... .......... 76% 54.5M 1s 2024-09-16T12:22:58.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:128250K .......... .......... .......... .......... .......... 76% 23.4M 1s 2024-09-16T12:22:58.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:128300K .......... .......... .......... .......... .......... 76% 36.1M 1s 2024-09-16T12:22:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:128350K .......... .......... .......... .......... .......... 76% 24.0M 1s 2024-09-16T12:22:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:128400K .......... .......... .......... .......... .......... 76% 24.8M 1s 2024-09-16T12:22:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:128450K .......... .......... .......... .......... .......... 76% 45.2M 1s 2024-09-16T12:22:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:128500K .......... .......... .......... .......... .......... 76% 25.9M 1s 2024-09-16T12:22:58.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:128550K .......... .......... .......... .......... .......... 76% 34.9M 1s 2024-09-16T12:22:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:128600K .......... .......... .......... .......... .......... 76% 23.5M 1s 2024-09-16T12:22:58.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:128650K .......... .......... .......... .......... .......... 76% 53.6M 1s 2024-09-16T12:22:58.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:128700K .......... .......... .......... .......... .......... 76% 25.6M 1s 2024-09-16T12:22:58.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:128750K .......... .......... .......... .......... .......... 76% 24.6M 1s 2024-09-16T12:22:58.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:128800K .......... .......... .......... .......... .......... 76% 23.0M 1s 2024-09-16T12:22:58.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:128850K .......... .......... .......... .......... .......... 76% 30.9M 1s 2024-09-16T12:22:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:128900K .......... .......... .......... .......... .......... 76% 26.6M 1s 2024-09-16T12:22:58.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:128950K .......... .......... .......... .......... .......... 76% 53.6M 1s 2024-09-16T12:22:58.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:129000K .......... .......... .......... .......... .......... 76% 28.5M 1s 2024-09-16T12:22:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:129050K .......... .......... .......... .......... .......... 76% 31.7M 1s 2024-09-16T12:22:58.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:129100K .......... .......... .......... .......... .......... 76% 29.7M 1s 2024-09-16T12:22:58.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:129150K .......... .......... .......... .......... .......... 76% 18.6M 1s 2024-09-16T12:22:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:129200K .......... .......... .......... .......... .......... 76% 25.7M 1s 2024-09-16T12:22:58.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:129250K .......... .......... .......... .......... .......... 76% 54.2M 1s 2024-09-16T12:22:58.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:129300K .......... .......... .......... .......... .......... 76% 23.4M 1s 2024-09-16T12:22:58.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:129350K .......... .......... .......... .......... .......... 76% 36.9M 1s 2024-09-16T12:22:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:129400K .......... .......... .......... .......... .......... 76% 21.4M 1s 2024-09-16T12:22:58.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:129450K .......... .......... .......... .......... .......... 76% 83.6M 1s 2024-09-16T12:22:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:129500K .......... .......... .......... .......... .......... 76% 22.9M 1s 2024-09-16T12:22:58.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:129550K .......... .......... .......... .......... .......... 76% 23.6M 1s 2024-09-16T12:22:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:129600K .......... .......... .......... .......... .......... 76% 38.1M 1s 2024-09-16T12:22:58.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:129650K .......... .......... .......... .......... .......... 76% 26.1M 1s 2024-09-16T12:22:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:129700K .......... .......... .......... .......... .......... 76% 21.5M 1s 2024-09-16T12:22:58.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:129750K .......... .......... .......... .......... .......... 77% 51.1M 1s 2024-09-16T12:22:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:129800K .......... .......... .......... .......... .......... 77% 23.6M 1s 2024-09-16T12:22:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:129850K .......... .......... .......... .......... .......... 77% 34.9M 1s 2024-09-16T12:22:58.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:129900K .......... .......... .......... .......... .......... 77% 25.4M 1s 2024-09-16T12:22:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:129950K .......... .......... .......... .......... .......... 77% 24.4M 1s 2024-09-16T12:22:58.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:130000K .......... .......... .......... .......... .......... 77% 45.1M 1s 2024-09-16T12:22:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:130050K .......... .......... .......... .......... .......... 77% 25.1M 1s 2024-09-16T12:22:58.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:130100K .......... .......... .......... .......... .......... 77% 38.0M 1s 2024-09-16T12:22:58.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:130150K .......... .......... .......... .......... .......... 77% 25.0M 1s 2024-09-16T12:22:58.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:130200K .......... .......... .......... .......... .......... 77% 49.0M 1s 2024-09-16T12:22:58.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:130250K .......... .......... .......... .......... .......... 77% 22.7M 1s 2024-09-16T12:22:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:130300K .......... .......... .......... .......... .......... 77% 32.8M 1s 2024-09-16T12:22:58.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:130350K .......... .......... .......... .......... .......... 77% 25.9M 1s 2024-09-16T12:22:58.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:130400K .......... .......... .......... .......... .......... 77% 23.7M 1s 2024-09-16T12:22:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:130450K .......... .......... .......... .......... .......... 77% 24.7M 1s 2024-09-16T12:22:58.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:130500K .......... .......... .......... .......... .......... 77% 41.1M 1s 2024-09-16T12:22:58.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:130550K .......... .......... .......... .......... .......... 77% 26.0M 1s 2024-09-16T12:22:58.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:130600K .......... .......... .......... .......... .......... 77% 33.4M 1s 2024-09-16T12:22:58.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:130650K .......... .......... .......... .......... .......... 77% 25.5M 1s 2024-09-16T12:22:58.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:130700K .......... .......... .......... .......... .......... 77% 48.1M 1s 2024-09-16T12:22:58.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:130750K .......... .......... .......... .......... .......... 77% 23.9M 1s 2024-09-16T12:22:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:130800K .......... .......... .......... .......... .......... 77% 24.7M 1s 2024-09-16T12:22:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:130850K .......... .......... .......... .......... .......... 77% 36.8M 1s 2024-09-16T12:22:58.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:130900K .......... .......... .......... .......... .......... 77% 24.1M 1s 2024-09-16T12:22:58.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:130950K .......... .......... .......... .......... .......... 77% 26.1M 1s 2024-09-16T12:22:58.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:131000K .......... .......... .......... .......... .......... 77% 39.7M 1s 2024-09-16T12:22:58.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:131050K .......... .......... .......... .......... .......... 77% 26.8M 1s 2024-09-16T12:22:58.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:131100K .......... .......... .......... .......... .......... 77% 33.9M 1s 2024-09-16T12:22:58.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:131150K .......... .......... .......... .......... .......... 77% 23.4M 1s 2024-09-16T12:22:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:131200K .......... .......... .......... .......... .......... 77% 25.8M 1s 2024-09-16T12:22:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:131250K .......... .......... .......... .......... .......... 77% 48.2M 1s 2024-09-16T12:22:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:131300K .......... .......... .......... .......... .......... 77% 24.1M 1s 2024-09-16T12:22:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:131350K .......... .......... .......... .......... .......... 77% 25.0M 1s 2024-09-16T12:22:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:131400K .......... .......... .......... .......... .......... 77% 37.3M 1s 2024-09-16T12:22:58.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:131450K .......... .......... .......... .......... .......... 78% 25.7M 1s 2024-09-16T12:22:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:131500K .......... .......... .......... .......... .......... 78% 37.6M 1s 2024-09-16T12:22:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:131550K .......... .......... .......... .......... .......... 78% 22.5M 1s 2024-09-16T12:22:58.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:131600K .......... .......... .......... .......... .......... 78% 25.9M 1s 2024-09-16T12:22:58.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:131650K .......... .......... .......... .......... .......... 78% 36.3M 1s 2024-09-16T12:22:58.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:131700K .......... .......... .......... .......... .......... 78% 34.0M 1s 2024-09-16T12:22:58.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:131750K .......... .......... .......... .......... .......... 78% 35.1M 1s 2024-09-16T12:22:58.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:131800K .......... .......... .......... .......... .......... 78% 23.2M 1s 2024-09-16T12:22:58.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:131850K .......... .......... .......... .......... .......... 78% 36.2M 1s 2024-09-16T12:22:58.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:131900K .......... .......... .......... .......... .......... 78% 24.0M 1s 2024-09-16T12:22:58.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:131950K .......... .......... .......... .......... .......... 78% 23.4M 1s 2024-09-16T12:22:58.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:132000K .......... .......... .......... .......... .......... 78% 31.5M 1s 2024-09-16T12:22:58.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:132050K .......... .......... .......... .......... .......... 78% 35.9M 1s 2024-09-16T12:22:58.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:132100K .......... .......... .......... .......... .......... 78% 32.9M 1s 2024-09-16T12:22:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:132150K .......... .......... .......... .......... .......... 78% 23.0M 1s 2024-09-16T12:22:58.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:132200K .......... .......... .......... .......... .......... 78% 25.4M 1s 2024-09-16T12:22:58.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:132250K .......... .......... .......... .......... .......... 78% 46.4M 1s 2024-09-16T12:22:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:132300K .......... .......... .......... .......... .......... 78% 25.1M 1s 2024-09-16T12:22:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:132350K .......... .......... .......... .......... .......... 78% 27.1M 1s 2024-09-16T12:22:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:132400K .......... .......... .......... .......... .......... 78% 22.2M 1s 2024-09-16T12:22:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:132450K .......... .......... .......... .......... .......... 78% 38.7M 1s 2024-09-16T12:22:58.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:132500K .......... .......... .......... .......... .......... 78% 46.8M 1s 2024-09-16T12:22:58.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:132550K .......... .......... .......... .......... .......... 78% 23.6M 1s 2024-09-16T12:22:58.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:132600K .......... .......... .......... .......... .......... 78% 36.0M 1s 2024-09-16T12:22:58.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:132650K .......... .......... .......... .......... .......... 78% 28.0M 1s 2024-09-16T12:22:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:132700K .......... .......... .......... .......... .......... 78% 22.0M 1s 2024-09-16T12:22:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:132750K .......... .......... .......... .......... .......... 78% 24.5M 1s 2024-09-16T12:22:58.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:132800K .......... .......... .......... .......... .......... 78% 47.2M 1s 2024-09-16T12:22:58.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:132850K .......... .......... .......... .......... .......... 78% 24.7M 1s 2024-09-16T12:22:58.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:132900K .......... .......... .......... .......... .......... 78% 38.7M 1s 2024-09-16T12:22:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:132950K .......... .......... .......... .......... .......... 78% 23.7M 1s 2024-09-16T12:22:58.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:133000K .......... .......... .......... .......... .......... 78% 43.3M 1s 2024-09-16T12:22:58.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:133050K .......... .......... .......... .......... .......... 78% 25.0M 1s 2024-09-16T12:22:58.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:133100K .......... .......... .......... .......... .......... 78% 24.3M 1s 2024-09-16T12:22:58.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:133150K .......... .......... .......... .......... .......... 79% 35.6M 1s 2024-09-16T12:22:58.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:133200K .......... .......... .......... .......... .......... 79% 24.3M 1s 2024-09-16T12:22:58.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:133250K .......... .......... .......... .......... .......... 79% 25.3M 1s 2024-09-16T12:22:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:133300K .......... .......... .......... .......... .......... 79% 43.9M 1s 2024-09-16T12:22:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:133350K .......... .......... .......... .......... .......... 79% 24.3M 1s 2024-09-16T12:22:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:133400K .......... .......... .......... .......... .......... 79% 34.8M 1s 2024-09-16T12:22:58.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:133450K .......... .......... .......... .......... .......... 79% 24.8M 1s 2024-09-16T12:22:58.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:133500K .......... .......... .......... .......... .......... 79% 49.5M 1s 2024-09-16T12:22:58.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:133550K .......... .......... .......... .......... .......... 79% 22.8M 1s 2024-09-16T12:22:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:133600K .......... .......... .......... .......... .......... 79% 26.0M 1s 2024-09-16T12:22:58.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:133650K .......... .......... .......... .......... .......... 79% 35.8M 1s 2024-09-16T12:22:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:133700K .......... .......... .......... .......... .......... 79% 22.1M 1s 2024-09-16T12:22:58.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:133750K .......... .......... .......... .......... .......... 79% 18.9M 1s 2024-09-16T12:22:58.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:133800K .......... .......... .......... .......... .......... 79% 37.6M 1s 2024-09-16T12:22:58.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:133850K .......... .......... .......... .......... .......... 79% 24.3M 1s 2024-09-16T12:22:58.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:133900K .......... .......... .......... .......... .......... 79% 37.6M 1s 2024-09-16T12:22:58.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:133950K .......... .......... .......... .......... .......... 79% 23.6M 1s 2024-09-16T12:22:58.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:134000K .......... .......... .......... .......... .......... 79% 24.9M 1s 2024-09-16T12:22:58.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:134050K .......... .......... .......... .......... .......... 79% 47.3M 1s 2024-09-16T12:22:58.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:134100K .......... .......... .......... .......... .......... 79% 25.3M 1s 2024-09-16T12:22:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:134150K .......... .......... .......... .......... .......... 79% 35.4M 1s 2024-09-16T12:22:58.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:134200K .......... .......... .......... .......... .......... 79% 25.3M 1s 2024-09-16T12:22:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:134250K .......... .......... .......... .......... .......... 79% 22.5M 1s 2024-09-16T12:22:58.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:134300K .......... .......... .......... .......... .......... 79% 57.0M 1s 2024-09-16T12:22:58.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:134350K .......... .......... .......... .......... .......... 79% 24.2M 1s 2024-09-16T12:22:58.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:134400K .......... .......... .......... .......... .......... 79% 23.4M 1s 2024-09-16T12:22:58.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:134450K .......... .......... .......... .......... .......... 79% 37.0M 1s 2024-09-16T12:22:58.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:134500K .......... .......... .......... .......... .......... 79% 25.0M 1s 2024-09-16T12:22:58.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:134550K .......... .......... .......... .......... .......... 79% 45.4M 1s 2024-09-16T12:22:58.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:134600K .......... .......... .......... .......... .......... 79% 24.2M 1s 2024-09-16T12:22:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:134650K .......... .......... .......... .......... .......... 79% 38.8M 1s 2024-09-16T12:22:58.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:134700K .......... .......... .......... .......... .......... 79% 24.5M 1s 2024-09-16T12:22:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:134750K .......... .......... .......... .......... .......... 79% 25.1M 1s 2024-09-16T12:22:58.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:134800K .......... .......... .......... .......... .......... 79% 21.5M 1s 2024-09-16T12:22:58.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:134850K .......... .......... .......... .......... .......... 80% 54.5M 1s 2024-09-16T12:22:58.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:134900K .......... .......... .......... .......... .......... 80% 32.9M 1s 2024-09-16T12:22:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:134950K .......... .......... .......... .......... .......... 80% 24.1M 1s 2024-09-16T12:22:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:135000K .......... .......... .......... .......... .......... 80% 26.0M 1s 2024-09-16T12:22:58.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:135050K .......... .......... .......... .......... .......... 80% 46.2M 1s 2024-09-16T12:22:58.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:135100K .......... .......... .......... .......... .......... 80% 24.1M 1s 2024-09-16T12:22:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:135150K .......... .......... .......... .......... .......... 80% 24.4M 1s 2024-09-16T12:22:58.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:135200K .......... .......... .......... .......... .......... 80% 39.9M 1s 2024-09-16T12:22:58.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:135250K .......... .......... .......... .......... .......... 80% 23.7M 1s 2024-09-16T12:22:58.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:135300K .......... .......... .......... .......... .......... 80% 45.8M 1s 2024-09-16T12:22:58.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:135350K .......... .......... .......... .......... .......... 80% 24.9M 1s 2024-09-16T12:22:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:135400K .......... .......... .......... .......... .......... 80% 35.3M 1s 2024-09-16T12:22:58.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:135450K .......... .......... .......... .......... .......... 80% 23.8M 1s 2024-09-16T12:22:58.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:135500K .......... .......... .......... .......... .......... 80% 25.1M 1s 2024-09-16T12:22:58.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:135550K .......... .......... .......... .......... .......... 80% 24.0M 1s 2024-09-16T12:22:58.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:135600K .......... .......... .......... .......... .......... 80% 46.5M 1s 2024-09-16T12:22:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:135650K .......... .......... .......... .......... .......... 80% 25.1M 1s 2024-09-16T12:22:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:135700K .......... .......... .......... .......... .......... 80% 35.1M 1s 2024-09-16T12:22:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:135750K .......... .......... .......... .......... .......... 80% 25.1M 1s 2024-09-16T12:22:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:135800K .......... .......... .......... .......... .......... 80% 48.6M 1s 2024-09-16T12:22:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:135850K .......... .......... .......... .......... .......... 80% 24.7M 1s 2024-09-16T12:22:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:135900K .......... .......... .......... .......... .......... 80% 36.8M 1s 2024-09-16T12:22:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:135950K .......... .......... .......... .......... .......... 80% 22.9M 1s 2024-09-16T12:22:58.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:136000K .......... .......... .......... .......... .......... 80% 25.4M 1s 2024-09-16T12:22:58.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:136050K .......... .......... .......... .......... .......... 80% 24.2M 1s 2024-09-16T12:22:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:136100K .......... .......... .......... .......... .......... 80% 48.1M 1s 2024-09-16T12:22:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:136150K .......... .......... .......... .......... .......... 80% 24.0M 1s 2024-09-16T12:22:58.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:136200K .......... .......... .......... .......... .......... 80% 39.4M 1s 2024-09-16T12:22:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:136250K .......... .......... .......... .......... .......... 80% 24.6M 1s 2024-09-16T12:22:58.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:136300K .......... .......... .......... .......... .......... 80% 44.6M 1s 2024-09-16T12:22:58.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:136350K .......... .......... .......... .......... .......... 80% 23.3M 1s 2024-09-16T12:22:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:136400K .......... .......... .......... .......... .......... 80% 26.4M 1s 2024-09-16T12:22:58.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:136450K .......... .......... .......... .......... .......... 80% 39.1M 1s 2024-09-16T12:22:58.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:136500K .......... .......... .......... .......... .......... 81% 24.0M 1s 2024-09-16T12:22:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:136550K .......... .......... .......... .......... .......... 81% 22.0M 1s 2024-09-16T12:22:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:136600K .......... .......... .......... .......... .......... 81% 53.7M 1s 2024-09-16T12:22:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:136650K .......... .......... .......... .......... .......... 81% 33.8M 1s 2024-09-16T12:22:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:136700K .......... .......... .......... .......... .......... 81% 26.2M 1s 2024-09-16T12:22:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:136750K .......... .......... .......... .......... .......... 81% 23.6M 1s 2024-09-16T12:22:58.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:136800K .......... .......... .......... .......... .......... 81% 25.8M 1s 2024-09-16T12:22:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:136850K .......... .......... .......... .......... .......... 81% 42.7M 1s 2024-09-16T12:22:58.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:136900K .......... .......... .......... .......... .......... 81% 25.3M 1s 2024-09-16T12:22:58.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:136950K .......... .......... .......... .......... .......... 81% 36.5M 1s 2024-09-16T12:22:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:137000K .......... .......... .......... .......... .......... 81% 25.0M 1s 2024-09-16T12:22:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:137050K .......... .......... .......... .......... .......... 81% 45.2M 1s 2024-09-16T12:22:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:137100K .......... .......... .......... .......... .......... 81% 24.0M 1s 2024-09-16T12:22:58.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:137150K .......... .......... .......... .......... .......... 81% 25.1M 1s 2024-09-16T12:22:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:137200K .......... .......... .......... .......... .......... 81% 23.1M 1s 2024-09-16T12:22:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:137250K .......... .......... .......... .......... .......... 81% 37.0M 1s 2024-09-16T12:22:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:137300K .......... .......... .......... .......... .......... 81% 25.6M 1s 2024-09-16T12:22:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:137350K .......... .......... .......... .......... .......... 81% 45.1M 1s 2024-09-16T12:22:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:137400K .......... .......... .......... .......... .......... 81% 24.3M 1s 2024-09-16T12:22:58.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:137450K .......... .......... .......... .......... .......... 81% 39.8M 1s 2024-09-16T12:22:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:137500K .......... .......... .......... .......... .......... 81% 24.7M 1s 2024-09-16T12:22:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:137550K .......... .......... .......... .......... .......... 81% 24.0M 1s 2024-09-16T12:22:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:137600K .......... .......... .......... .......... .......... 81% 45.2M 1s 2024-09-16T12:22:58.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:137650K .......... .......... .......... .......... .......... 81% 17.0M 1s 2024-09-16T12:22:58.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:137700K .......... .......... .......... .......... .......... 81% 33.8M 1s 2024-09-16T12:22:58.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:137750K .......... .......... .......... .......... .......... 81% 24.0M 1s 2024-09-16T12:22:58.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:137800K .......... .......... .......... .......... .......... 81% 25.0M 1s 2024-09-16T12:22:58.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:137850K .......... .......... .......... .......... .......... 81% 42.6M 1s 2024-09-16T12:22:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:137900K .......... .......... .......... .......... .......... 81% 24.6M 1s 2024-09-16T12:22:58.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:137950K .......... .......... .......... .......... .......... 81% 24.1M 1s 2024-09-16T12:22:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:138000K .......... .......... .......... .......... .......... 81% 37.9M 1s 2024-09-16T12:22:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:138050K .......... .......... .......... .......... .......... 81% 23.9M 1s 2024-09-16T12:22:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:138100K .......... .......... .......... .......... .......... 81% 59.9M 1s 2024-09-16T12:22:58.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:138150K .......... .......... .......... .......... .......... 81% 22.2M 1s 2024-09-16T12:22:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:138200K .......... .......... .......... .......... .......... 82% 38.9M 1s 2024-09-16T12:22:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:138250K .......... .......... .......... .......... .......... 82% 23.6M 1s 2024-09-16T12:22:58.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:138300K .......... .......... .......... .......... .......... 82% 23.3M 1s 2024-09-16T12:22:58.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:138350K .......... .......... .......... .......... .......... 82% 25.5M 1s 2024-09-16T12:22:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:138400K .......... .......... .......... .......... .......... 82% 46.0M 1s 2024-09-16T12:22:58.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:138450K .......... .......... .......... .......... .......... 82% 31.4M 1s 2024-09-16T12:22:58.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:138500K .......... .......... .......... .......... .......... 82% 27.4M 1s 2024-09-16T12:22:58.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:138550K .......... .......... .......... .......... .......... 82% 24.6M 1s 2024-09-16T12:22:58.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:138600K .......... .......... .......... .......... .......... 82% 46.2M 1s 2024-09-16T12:22:58.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:138650K .......... .......... .......... .......... .......... 82% 25.5M 1s 2024-09-16T12:22:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:138700K .......... .......... .......... .......... .......... 82% 35.8M 1s 2024-09-16T12:22:58.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:138750K .......... .......... .......... .......... .......... 82% 24.5M 1s 2024-09-16T12:22:58.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:138800K .......... .......... .......... .......... .......... 82% 24.5M 1s 2024-09-16T12:22:58.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:138850K .......... .......... .......... .......... .......... 82% 23.4M 1s 2024-09-16T12:22:58.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:138900K .......... .......... .......... .......... .......... 82% 59.0M 1s 2024-09-16T12:22:58.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:138950K .......... .......... .......... .......... .......... 82% 22.3M 1s 2024-09-16T12:22:58.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:139000K .......... .......... .......... .......... .......... 82% 36.5M 1s 2024-09-16T12:22:58.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:139050K .......... .......... .......... .......... .......... 82% 24.7M 1s 2024-09-16T12:22:58.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:139100K .......... .......... .......... .......... .......... 82% 46.4M 1s 2024-09-16T12:22:58.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:139150K .......... .......... .......... .......... .......... 82% 24.5M 1s 2024-09-16T12:22:58.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:139200K .......... .......... .......... .......... .......... 82% 24.8M 1s 2024-09-16T12:22:58.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:139250K .......... .......... .......... .......... .......... 82% 35.8M 1s 2024-09-16T12:22:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:139300K .......... .......... .......... .......... .......... 82% 25.7M 1s 2024-09-16T12:22:58.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:139350K .......... .......... .......... .......... .......... 82% 21.9M 1s 2024-09-16T12:22:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:139400K .......... .......... .......... .......... .......... 82% 58.0M 1s 2024-09-16T12:22:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:139450K .......... .......... .......... .......... .......... 82% 34.7M 1s 2024-09-16T12:22:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:139500K .......... .......... .......... .......... .......... 82% 23.8M 1s 2024-09-16T12:22:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:139550K .......... .......... .......... .......... .......... 82% 24.1M 1s 2024-09-16T12:22:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:139600K .......... .......... .......... .......... .......... 82% 25.2M 1s 2024-09-16T12:22:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:139650K .......... .......... .......... .......... .......... 82% 44.5M 1s 2024-09-16T12:22:58.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:139700K .......... .......... .......... .......... .......... 82% 25.9M 1s 2024-09-16T12:22:58.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:139750K .......... .......... .......... .......... .......... 82% 35.6M 1s 2024-09-16T12:22:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:139800K .......... .......... .......... .......... .......... 82% 25.7M 1s 2024-09-16T12:22:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:139850K .......... .......... .......... .......... .......... 82% 44.1M 1s 2024-09-16T12:22:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:139900K .......... .......... .......... .......... .......... 83% 24.7M 1s 2024-09-16T12:22:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:139950K .......... .......... .......... .......... .......... 83% 24.4M 1s 2024-09-16T12:22:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:140000K .......... .......... .......... .......... .......... 83% 23.6M 1s 2024-09-16T12:22:58.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:140050K .......... .......... .......... .......... .......... 83% 39.7M 1s 2024-09-16T12:22:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:140100K .......... .......... .......... .......... .......... 83% 25.3M 1s 2024-09-16T12:22:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:140150K .......... .......... .......... .......... .......... 83% 38.5M 1s 2024-09-16T12:22:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:140200K .......... .......... .......... .......... .......... 83% 24.2M 1s 2024-09-16T12:22:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:140250K .......... .......... .......... .......... .......... 83% 40.5M 1s 2024-09-16T12:22:58.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:140300K .......... .......... .......... .......... .......... 83% 25.2M 1s 2024-09-16T12:22:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:140350K .......... .......... .......... .......... .......... 83% 23.0M 1s 2024-09-16T12:22:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:140400K .......... .......... .......... .......... .......... 83% 46.8M 1s 2024-09-16T12:22:58.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:140450K .......... .......... .......... .......... .......... 83% 24.5M 1s 2024-09-16T12:22:58.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:140500K .......... .......... .......... .......... .......... 83% 33.6M 1s 2024-09-16T12:22:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:140550K .......... .......... .......... .......... .......... 83% 25.0M 1s 2024-09-16T12:22:58.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:140600K .......... .......... .......... .......... .......... 83% 25.3M 1s 2024-09-16T12:22:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:140650K .......... .......... .......... .......... .......... 83% 43.7M 1s 2024-09-16T12:22:58.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:140700K .......... .......... .......... .......... .......... 83% 25.6M 1s 2024-09-16T12:22:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:140750K .......... .......... .......... .......... .......... 83% 24.2M 1s 2024-09-16T12:22:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:140800K .......... .......... .......... .......... .......... 83% 37.5M 1s 2024-09-16T12:22:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:140850K .......... .......... .......... .......... .......... 83% 25.3M 1s 2024-09-16T12:22:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:140900K .......... .......... .......... .......... .......... 83% 42.4M 1s 2024-09-16T12:22:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:140950K .......... .......... .......... .......... .......... 83% 26.2M 1s 2024-09-16T12:22:58.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:141000K .......... .......... .......... .......... .......... 83% 35.7M 1s 2024-09-16T12:22:58.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:141050K .......... .......... .......... .......... .......... 83% 22.5M 1s 2024-09-16T12:22:58.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:141100K .......... .......... .......... .......... .......... 83% 24.9M 1s 2024-09-16T12:22:58.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:141150K .......... .......... .......... .......... .......... 83% 25.1M 1s 2024-09-16T12:22:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:141200K .......... .......... .......... .......... .......... 83% 42.4M 1s 2024-09-16T12:22:58.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:141250K .......... .......... .......... .......... .......... 83% 25.5M 1s 2024-09-16T12:22:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:141300K .......... .......... .......... .......... .......... 83% 46.9M 1s 2024-09-16T12:22:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:141350K .......... .......... .......... .......... .......... 83% 21.2M 1s 2024-09-16T12:22:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:141400K .......... .......... .......... .......... .......... 83% 48.6M 1s 2024-09-16T12:22:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:141450K .......... .......... .......... .......... .......... 83% 23.4M 1s 2024-09-16T12:22:58.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:141500K .......... .......... .......... .......... .......... 83% 41.8M 1s 2024-09-16T12:22:58.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:141550K .......... .......... .......... .......... .......... 84% 20.2M 1s 2024-09-16T12:22:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:141600K .......... .......... .......... .......... .......... 84% 20.1M 1s 2024-09-16T12:22:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:141650K .......... .......... .......... .......... .......... 84% 27.4M 1s 2024-09-16T12:22:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:141700K .......... .......... .......... .......... .......... 84% 39.1M 1s 2024-09-16T12:22:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:141750K .......... .......... .......... .......... .......... 84% 23.9M 1s 2024-09-16T12:22:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:141800K .......... .......... .......... .......... .......... 84% 38.3M 1s 2024-09-16T12:22:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:141850K .......... .......... .......... .......... .......... 84% 24.2M 1s 2024-09-16T12:22:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:141900K .......... .......... .......... .......... .......... 84% 51.5M 1s 2024-09-16T12:22:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:141950K .......... .......... .......... .......... .......... 84% 23.2M 1s 2024-09-16T12:22:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:142000K .......... .......... .......... .......... .......... 84% 25.4M 1s 2024-09-16T12:22:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:142050K .......... .......... .......... .......... .......... 84% 37.7M 1s 2024-09-16T12:22:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:142100K .......... .......... .......... .......... .......... 84% 24.9M 1s 2024-09-16T12:22:58.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:142150K .......... .......... .......... .......... .......... 84% 45.2M 1s 2024-09-16T12:22:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:142200K .......... .......... .......... .......... .......... 84% 25.3M 1s 2024-09-16T12:22:58.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:142250K .......... .......... .......... .......... .......... 84% 32.1M 1s 2024-09-16T12:22:58.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:142300K .......... .......... .......... .......... .......... 84% 31.0M 1s 2024-09-16T12:22:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:142350K .......... .......... .......... .......... .......... 84% 20.7M 1s 2024-09-16T12:22:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:142400K .......... .......... .......... .......... .......... 84% 22.6M 1s 2024-09-16T12:22:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:142450K .......... .......... .......... .......... .......... 84% 60.9M 1s 2024-09-16T12:22:58.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:142500K .......... .......... .......... .......... .......... 84% 23.6M 1s 2024-09-16T12:22:58.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:142550K .......... .......... .......... .......... .......... 84% 34.5M 1s 2024-09-16T12:22:58.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:142600K .......... .......... .......... .......... .......... 84% 25.0M 1s 2024-09-16T12:22:58.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:142650K .......... .......... .......... .......... .......... 84% 45.8M 1s 2024-09-16T12:22:58.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:142700K .......... .......... .......... .......... .......... 84% 25.5M 1s 2024-09-16T12:22:58.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:142750K .......... .......... .......... .......... .......... 84% 23.2M 1s 2024-09-16T12:22:58.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:142800K .......... .......... .......... .......... .......... 84% 24.0M 1s 2024-09-16T12:22:58.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:142850K .......... .......... .......... .......... .......... 84% 35.8M 1s 2024-09-16T12:22:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:142900K .......... .......... .......... .......... .......... 84% 25.0M 1s 2024-09-16T12:22:58.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:142950K .......... .......... .......... .......... .......... 84% 45.9M 1s 2024-09-16T12:22:58.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:143000K .......... .......... .......... .......... .......... 84% 25.7M 1s 2024-09-16T12:22:58.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:143050K .......... .......... .......... .......... .......... 84% 35.6M 1s 2024-09-16T12:22:58.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:143100K .......... .......... .......... .......... .......... 84% 24.3M 1s 2024-09-16T12:22:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:143150K .......... .......... .......... .......... .......... 84% 24.9M 1s 2024-09-16T12:22:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:143200K .......... .......... .......... .......... .......... 84% 46.2M 1s 2024-09-16T12:22:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:143250K .......... .......... .......... .......... .......... 85% 24.7M 1s 2024-09-16T12:22:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:143300K .......... .......... .......... .......... .......... 85% 36.2M 1s 2024-09-16T12:22:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:143350K .......... .......... .......... .......... .......... 85% 22.7M 1s 2024-09-16T12:22:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:143400K .......... .......... .......... .......... .......... 85% 24.3M 1s 2024-09-16T12:22:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:143450K .......... .......... .......... .......... .......... 85% 61.9M 1s 2024-09-16T12:22:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:143500K .......... .......... .......... .......... .......... 85% 21.9M 1s 2024-09-16T12:22:58.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:143550K .......... .......... .......... .......... .......... 85% 31.1M 1s 2024-09-16T12:22:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:143600K .......... .......... .......... .......... .......... 85% 29.6M 1s 2024-09-16T12:22:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:143650K .......... .......... .......... .......... .......... 85% 32.3M 1s 2024-09-16T12:22:58.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:143700K .......... .......... .......... .......... .......... 85% 31.8M 1s 2024-09-16T12:22:58.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:143750K .......... .......... .......... .......... .......... 85% 23.8M 1s 2024-09-16T12:22:58.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:143800K .......... .......... .......... .......... .......... 85% 39.1M 1s 2024-09-16T12:22:58.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:143850K .......... .......... .......... .......... .......... 85% 30.3M 1s 2024-09-16T12:22:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:143900K .......... .......... .......... .......... .......... 85% 20.0M 1s 2024-09-16T12:22:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:143950K .......... .......... .......... .......... .......... 85% 24.3M 1s 2024-09-16T12:22:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:144000K .......... .......... .......... .......... .......... 85% 44.9M 1s 2024-09-16T12:22:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:144050K .......... .......... .......... .......... .......... 85% 28.6M 1s 2024-09-16T12:22:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:144100K .......... .......... .......... .......... .......... 85% 30.2M 1s 2024-09-16T12:22:58.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:144150K .......... .......... .......... .......... .......... 85% 26.5M 1s 2024-09-16T12:22:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:144200K .......... .......... .......... .......... .......... 85% 43.2M 1s 2024-09-16T12:22:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:144250K .......... .......... .......... .......... .......... 85% 23.8M 1s 2024-09-16T12:22:58.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:144300K .......... .......... .......... .......... .......... 85% 40.0M 1s 2024-09-16T12:22:58.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:144350K .......... .......... .......... .......... .......... 85% 24.6M 1s 2024-09-16T12:22:58.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:144400K .......... .......... .......... .......... .......... 85% 28.3M 1s 2024-09-16T12:22:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:144450K .......... .......... .......... .......... .......... 85% 22.0M 1s 2024-09-16T12:22:58.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:144500K .......... .......... .......... .......... .......... 85% 50.6M 1s 2024-09-16T12:22:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:144550K .......... .......... .......... .......... .......... 85% 22.4M 1s 2024-09-16T12:22:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:144600K .......... .......... .......... .......... .......... 85% 36.2M 1s 2024-09-16T12:22:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:144650K .......... .......... .......... .......... .......... 85% 25.5M 1s 2024-09-16T12:22:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:144700K .......... .......... .......... .......... .......... 85% 44.2M 1s 2024-09-16T12:22:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:144750K .......... .......... .......... .......... .......... 85% 22.5M 1s 2024-09-16T12:22:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:144800K .......... .......... .......... .......... .......... 85% 25.5M 1s 2024-09-16T12:22:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:144850K .......... .......... .......... .......... .......... 85% 36.7M 1s 2024-09-16T12:22:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:144900K .......... .......... .......... .......... .......... 85% 25.7M 1s 2024-09-16T12:22:58.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:144950K .......... .......... .......... .......... .......... 86% 43.9M 1s 2024-09-16T12:22:58.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:145000K .......... .......... .......... .......... .......... 86% 26.9M 1s 2024-09-16T12:22:58.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:145050K .......... .......... .......... .......... .......... 86% 31.4M 1s 2024-09-16T12:22:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:145100K .......... .......... .......... .......... .......... 86% 29.5M 1s 2024-09-16T12:22:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:145150K .......... .......... .......... .......... .......... 86% 21.0M 1s 2024-09-16T12:22:58.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:145200K .......... .......... .......... .......... .......... 86% 30.5M 1s 2024-09-16T12:22:58.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:145250K .......... .......... .......... .......... .......... 86% 38.4M 1s 2024-09-16T12:22:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:145300K .......... .......... .......... .......... .......... 86% 22.6M 1s 2024-09-16T12:22:58.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:145350K .......... .......... .......... .......... .......... 86% 35.8M 1s 2024-09-16T12:22:58.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:145400K .......... .......... .......... .......... .......... 86% 24.7M 1s 2024-09-16T12:22:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:145450K .......... .......... .......... .......... .......... 86% 46.5M 1s 2024-09-16T12:22:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:145500K .......... .......... .......... .......... .......... 86% 24.7M 1s 2024-09-16T12:22:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:145550K .......... .......... .......... .......... .......... 86% 24.3M 1s 2024-09-16T12:22:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:145600K .......... .......... .......... .......... .......... 86% 35.4M 1s 2024-09-16T12:22:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:145650K .......... .......... .......... .......... .......... 86% 24.9M 1s 2024-09-16T12:22:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:145700K .......... .......... .......... .......... .......... 86% 24.6M 1s 2024-09-16T12:22:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:145750K .......... .......... .......... .......... .......... 86% 39.6M 1s 2024-09-16T12:22:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:145800K .......... .......... .......... .......... .......... 86% 24.7M 1s 2024-09-16T12:22:58.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:145850K .......... .......... .......... .......... .......... 86% 37.5M 1s 2024-09-16T12:22:58.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:145900K .......... .......... .......... .......... .......... 86% 24.3M 1s 2024-09-16T12:22:58.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:145950K .......... .......... .......... .......... .......... 86% 22.5M 1s 2024-09-16T12:22:58.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:146000K .......... .......... .......... .......... .......... 86% 44.9M 1s 2024-09-16T12:22:58.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:146050K .......... .......... .......... .......... .......... 86% 26.3M 1s 2024-09-16T12:22:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:146100K .......... .......... .......... .......... .......... 86% 33.9M 1s 2024-09-16T12:22:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:146150K .......... .......... .......... .......... .......... 86% 24.1M 1s 2024-09-16T12:22:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:146200K .......... .......... .......... .......... .......... 86% 23.7M 1s 2024-09-16T12:22:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:146250K .......... .......... .......... .......... .......... 86% 55.6M 1s 2024-09-16T12:22:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:146300K .......... .......... .......... .......... .......... 86% 24.2M 1s 2024-09-16T12:22:58.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:146350K .......... .......... .......... .......... .......... 86% 23.5M 1s 2024-09-16T12:22:58.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:146400K .......... .......... .......... .......... .......... 86% 34.7M 1s 2024-09-16T12:22:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:146450K .......... .......... .......... .......... .......... 86% 25.6M 1s 2024-09-16T12:22:58.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:146500K .......... .......... .......... .......... .......... 86% 61.7M 1s 2024-09-16T12:22:58.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:146550K .......... .......... .......... .......... .......... 86% 21.5M 1s 2024-09-16T12:22:58.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:146600K .......... .......... .......... .......... .......... 86% 40.5M 1s 2024-09-16T12:22:58.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:146650K .......... .......... .......... .......... .......... 87% 23.8M 1s 2024-09-16T12:22:58.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:146700K .......... .......... .......... .......... .......... 87% 29.4M 1s 2024-09-16T12:22:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:146750K .......... .......... .......... .......... .......... 87% 20.2M 1s 2024-09-16T12:22:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:146800K .......... .......... .......... .......... .......... 87% 49.4M 1s 2024-09-16T12:22:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:146850K .......... .......... .......... .......... .......... 87% 23.4M 1s 2024-09-16T12:22:58.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:146900K .......... .......... .......... .......... .......... 87% 36.2M 1s 2024-09-16T12:22:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:146950K .......... .......... .......... .......... .......... 87% 24.6M 1s 2024-09-16T12:22:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:147000K .......... .......... .......... .......... .......... 87% 46.1M 1s 2024-09-16T12:22:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:147050K .......... .......... .......... .......... .......... 87% 24.4M 1s 2024-09-16T12:22:58.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:147100K .......... .......... .......... .......... .......... 87% 38.8M 1s 2024-09-16T12:22:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:147150K .......... .......... .......... .......... .......... 87% 23.4M 1s 2024-09-16T12:22:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:147200K .......... .......... .......... .......... .......... 87% 25.6M 1s 2024-09-16T12:22:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:147250K .......... .......... .......... .......... .......... 87% 24.2M 1s 2024-09-16T12:22:58.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:147300K .......... .......... .......... .......... .......... 87% 45.8M 1s 2024-09-16T12:22:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:147350K .......... .......... .......... .......... .......... 87% 31.4M 1s 2024-09-16T12:22:58.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:147400K .......... .......... .......... .......... .......... 87% 25.6M 1s 2024-09-16T12:22:58.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:147450K .......... .......... .......... .......... .......... 87% 26.2M 1s 2024-09-16T12:22:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:147500K .......... .......... .......... .......... .......... 87% 44.4M 1s 2024-09-16T12:22:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:147550K .......... .......... .......... .......... .......... 87% 22.5M 1s 2024-09-16T12:22:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:147600K .......... .......... .......... .......... .......... 87% 26.3M 1s 2024-09-16T12:22:58.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:147650K .......... .......... .......... .......... .......... 87% 34.0M 1s 2024-09-16T12:22:58.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:147700K .......... .......... .......... .......... .......... 87% 25.3M 1s 2024-09-16T12:22:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:147750K .......... .......... .......... .......... .......... 87% 42.0M 1s 2024-09-16T12:22:58.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:147800K .......... .......... .......... .......... .......... 87% 25.7M 1s 2024-09-16T12:22:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:147850K .......... .......... .......... .......... .......... 87% 22.7M 1s 2024-09-16T12:22:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:147900K .......... .......... .......... .......... .......... 87% 38.7M 1s 2024-09-16T12:22:58.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:147950K .......... .......... .......... .......... .......... 87% 23.1M 1s 2024-09-16T12:22:58.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:148000K .......... .......... .......... .......... .......... 87% 26.3M 1s 2024-09-16T12:22:58.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:148050K .......... .......... .......... .......... .......... 87% 49.2M 1s 2024-09-16T12:22:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:148100K .......... .......... .......... .......... .......... 87% 23.9M 1s 2024-09-16T12:22:58.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:148150K .......... .......... .......... .......... .......... 87% 38.8M 1s 2024-09-16T12:22:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:148200K .......... .......... .......... .......... .......... 87% 24.0M 1s 2024-09-16T12:22:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:148250K .......... .......... .......... .......... .......... 87% 47.6M 1s 2024-09-16T12:22:58.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:148300K .......... .......... .......... .......... .......... 88% 25.1M 1s 2024-09-16T12:22:58.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:148350K .......... .......... .......... .......... .......... 88% 22.6M 1s 2024-09-16T12:22:58.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:148400K .......... .......... .......... .......... .......... 88% 39.5M 1s 2024-09-16T12:22:58.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:148450K .......... .......... .......... .......... .......... 88% 22.7M 1s 2024-09-16T12:22:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:148500K .......... .......... .......... .......... .......... 88% 25.9M 1s 2024-09-16T12:22:58.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:148550K .......... .......... .......... .......... .......... 88% 42.5M 1s 2024-09-16T12:22:58.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:148600K .......... .......... .......... .......... .......... 88% 24.6M 1s 2024-09-16T12:22:58.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:148650K .......... .......... .......... .......... .......... 88% 38.1M 1s 2024-09-16T12:22:58.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:148700K .......... .......... .......... .......... .......... 88% 24.2M 1s 2024-09-16T12:22:58.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:148750K .......... .......... .......... .......... .......... 88% 25.5M 1s 2024-09-16T12:22:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:148800K .......... .......... .......... .......... .......... 88% 46.8M 1s 2024-09-16T12:22:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:148850K .......... .......... .......... .......... .......... 88% 24.7M 1s 2024-09-16T12:22:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:148900K .......... .......... .......... .......... .......... 88% 34.5M 1s 2024-09-16T12:22:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:148950K .......... .......... .......... .......... .......... 88% 23.5M 1s 2024-09-16T12:22:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:149000K .......... .......... .......... .......... .......... 88% 23.7M 1s 2024-09-16T12:22:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:149050K .......... .......... .......... .......... .......... 88% 57.3M 1s 2024-09-16T12:22:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:149100K .......... .......... .......... .......... .......... 88% 28.4M 1s 2024-09-16T12:22:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:149150K .......... .......... .......... .......... .......... 88% 23.6M 1s 2024-09-16T12:22:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:149200K .......... .......... .......... .......... .......... 88% 30.7M 1s 2024-09-16T12:22:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:149250K .......... .......... .......... .......... .......... 88% 24.3M 1s 2024-09-16T12:22:58.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:149300K .......... .......... .......... .......... .......... 88% 60.0M 1s 2024-09-16T12:22:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:149350K .......... .......... .......... .......... .......... 88% 21.9M 1s 2024-09-16T12:22:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:149400K .......... .......... .......... .......... .......... 88% 34.9M 1s 2024-09-16T12:22:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:149450K .......... .......... .......... .......... .......... 88% 26.7M 1s 2024-09-16T12:22:58.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:149500K .......... .......... .......... .......... .......... 88% 47.9M 1s 2024-09-16T12:22:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:149550K .......... .......... .......... .......... .......... 88% 15.6M 1s 2024-09-16T12:22:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:149600K .......... .......... .......... .......... .......... 88% 52.3M 1s 2024-09-16T12:22:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:149650K .......... .......... .......... .......... .......... 88% 23.9M 1s 2024-09-16T12:22:58.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:149700K .......... .......... .......... .......... .......... 88% 38.4M 1s 2024-09-16T12:22:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:149750K .......... .......... .......... .......... .......... 88% 23.0M 1s 2024-09-16T12:22:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:149800K .......... .......... .......... .......... .......... 88% 39.0M 1s 2024-09-16T12:22:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:149850K .......... .......... .......... .......... .......... 88% 25.4M 1s 2024-09-16T12:22:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:149900K .......... .......... .......... .......... .......... 88% 36.2M 1s 2024-09-16T12:22:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:149950K .......... .......... .......... .......... .......... 88% 26.0M 1s 2024-09-16T12:22:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:150000K .......... .......... .......... .......... .......... 89% 25.1M 1s 2024-09-16T12:22:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:150050K .......... .......... .......... .......... .......... 89% 41.1M 1s 2024-09-16T12:22:58.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:150100K .......... .......... .......... .......... .......... 89% 24.1M 1s 2024-09-16T12:22:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:150150K .......... .......... .......... .......... .......... 89% 31.6M 1s 2024-09-16T12:22:58.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:150200K .......... .......... .......... .......... .......... 89% 28.2M 1s 2024-09-16T12:22:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:150250K .......... .......... .......... .......... .......... 89% 33.7M 1s 2024-09-16T12:22:58.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:150300K .......... .......... .......... .......... .......... 89% 32.1M 1s 2024-09-16T12:22:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:150350K .......... .......... .......... .......... .......... 89% 21.5M 1s 2024-09-16T12:22:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:150400K .......... .......... .......... .......... .......... 89% 36.4M 1s 2024-09-16T12:22:58.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:150450K .......... .......... .......... .......... .......... 89% 26.0M 1s 2024-09-16T12:22:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:150500K .......... .......... .......... .......... .......... 89% 26.0M 1s 2024-09-16T12:22:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:150550K .......... .......... .......... .......... .......... 89% 45.6M 1s 2024-09-16T12:22:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:150600K .......... .......... .......... .......... .......... 89% 23.8M 1s 2024-09-16T12:22:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:150650K .......... .......... .......... .......... .......... 89% 36.7M 1s 2024-09-16T12:22:58.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:150700K .......... .......... .......... .......... .......... 89% 23.0M 1s 2024-09-16T12:22:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:150750K .......... .......... .......... .......... .......... 89% 23.3M 1s 2024-09-16T12:22:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:150800K .......... .......... .......... .......... .......... 89% 27.7M 1s 2024-09-16T12:22:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:150850K .......... .......... .......... .......... .......... 89% 44.6M 1s 2024-09-16T12:22:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:150900K .......... .......... .......... .......... .......... 89% 20.1M 1s 2024-09-16T12:22:58.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:150950K .......... .......... .......... .......... .......... 89% 51.7M 1s 2024-09-16T12:22:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:151000K .......... .......... .......... .......... .......... 89% 26.4M 1s 2024-09-16T12:22:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:151050K .......... .......... .......... .......... .......... 89% 49.1M 1s 2024-09-16T12:22:58.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:151100K .......... .......... .......... .......... .......... 89% 24.0M 1s 2024-09-16T12:22:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:151150K .......... .......... .......... .......... .......... 89% 24.1M 1s 2024-09-16T12:22:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:151200K .......... .......... .......... .......... .......... 89% 35.2M 1s 2024-09-16T12:22:58.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:151250K .......... .......... .......... .......... .......... 89% 23.8M 1s 2024-09-16T12:22:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:151300K .......... .......... .......... .......... .......... 89% 25.1M 1s 2024-09-16T12:22:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:151350K .......... .......... .......... .......... .......... 89% 50.1M 1s 2024-09-16T12:22:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:151400K .......... .......... .......... .......... .......... 89% 23.3M 1s 2024-09-16T12:22:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:151450K .......... .......... .......... .......... .......... 89% 37.3M 1s 2024-09-16T12:22:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:151500K .......... .......... .......... .......... .......... 89% 24.6M 1s 2024-09-16T12:22:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:151550K .......... .......... .......... .......... .......... 89% 25.0M 1s 2024-09-16T12:22:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:151600K .......... .......... .......... .......... .......... 89% 42.4M 1s 2024-09-16T12:22:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:151650K .......... .......... .......... .......... .......... 89% 25.9M 1s 2024-09-16T12:22:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:151700K .......... .......... .......... .......... .......... 90% 37.4M 1s 2024-09-16T12:22:58.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:151750K .......... .......... .......... .......... .......... 90% 24.5M 1s 2024-09-16T12:22:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:151800K .......... .......... .......... .......... .......... 90% 23.5M 1s 2024-09-16T12:22:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:151850K .......... .......... .......... .......... .......... 90% 42.3M 1s 2024-09-16T12:22:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:151900K .......... .......... .......... .......... .......... 90% 24.9M 1s 2024-09-16T12:22:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:151950K .......... .......... .......... .......... .......... 90% 23.8M 1s 2024-09-16T12:22:58.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:152000K .......... .......... .......... .......... .......... 90% 38.3M 1s 2024-09-16T12:22:58.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:152050K .......... .......... .......... .......... .......... 90% 25.4M 1s 2024-09-16T12:22:58.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:152100K .......... .......... .......... .......... .......... 90% 45.7M 1s 2024-09-16T12:22:58.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:152150K .......... .......... .......... .......... .......... 90% 23.4M 1s 2024-09-16T12:22:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:152200K .......... .......... .......... .......... .......... 90% 40.6M 1s 2024-09-16T12:22:58.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:152250K .......... .......... .......... .......... .......... 90% 23.3M 1s 2024-09-16T12:22:58.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:152300K .......... .......... .......... .......... .......... 90% 49.3M 1s 2024-09-16T12:22:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:152350K .......... .......... .......... .......... .......... 90% 16.2M 1s 2024-09-16T12:22:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:152400K .......... .......... .......... .......... .......... 90% 40.6M 1s 2024-09-16T12:22:58.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:152450K .......... .......... .......... .......... .......... 90% 33.5M 1s 2024-09-16T12:22:58.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:152500K .......... .......... .......... .......... .......... 90% 26.8M 1s 2024-09-16T12:22:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:152550K .......... .......... .......... .......... .......... 90% 25.0M 1s 2024-09-16T12:22:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:152600K .......... .......... .......... .......... .......... 90% 45.9M 1s 2024-09-16T12:22:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:152650K .......... .......... .......... .......... .......... 90% 24.7M 1s 2024-09-16T12:22:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:152700K .......... .......... .......... .......... .......... 90% 39.9M 1s 2024-09-16T12:22:58.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:152750K .......... .......... .......... .......... .......... 90% 23.7M 1s 2024-09-16T12:22:58.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:152800K .......... .......... .......... .......... .......... 90% 25.1M 1s 2024-09-16T12:22:58.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:152850K .......... .......... .......... .......... .......... 90% 39.8M 1s 2024-09-16T12:22:58.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:152900K .......... .......... .......... .......... .......... 90% 26.0M 1s 2024-09-16T12:22:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:152950K .......... .......... .......... .......... .......... 90% 32.2M 1s 2024-09-16T12:22:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:153000K .......... .......... .......... .......... .......... 90% 25.0M 1s 2024-09-16T12:22:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:153050K .......... .......... .......... .......... .......... 90% 25.3M 1s 2024-09-16T12:22:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:153100K .......... .......... .......... .......... .......... 90% 44.6M 1s 2024-09-16T12:22:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:153150K .......... .......... .......... .......... .......... 90% 25.1M 1s 2024-09-16T12:22:58.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:153200K .......... .......... .......... .......... .......... 90% 24.9M 1s 2024-09-16T12:22:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:153250K .......... .......... .......... .......... .......... 90% 37.1M 1s 2024-09-16T12:22:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:153300K .......... .......... .......... .......... .......... 90% 25.3M 1s 2024-09-16T12:22:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:153350K .......... .......... .......... .......... .......... 91% 44.1M 1s 2024-09-16T12:22:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:153400K .......... .......... .......... .......... .......... 91% 24.8M 1s 2024-09-16T12:22:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:153450K .......... .......... .......... .......... .......... 91% 37.4M 1s 2024-09-16T12:22:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:153500K .......... .......... .......... .......... .......... 91% 22.5M 1s 2024-09-16T12:22:58.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:153550K .......... .......... .......... .......... .......... 91% 23.5M 1s 2024-09-16T12:22:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:153600K .......... .......... .......... .......... .......... 91% 26.1M 1s 2024-09-16T12:22:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:153650K .......... .......... .......... .......... .......... 91% 48.0M 1s 2024-09-16T12:22:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:153700K .......... .......... .......... .......... .......... 91% 24.5M 1s 2024-09-16T12:22:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:153750K .......... .......... .......... .......... .......... 91% 36.0M 1s 2024-09-16T12:22:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:153800K .......... .......... .......... .......... .......... 91% 24.2M 1s 2024-09-16T12:22:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:153850K .......... .......... .......... .......... .......... 91% 42.4M 1s 2024-09-16T12:22:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:153900K .......... .......... .......... .......... .......... 91% 6.80M 1s 2024-09-16T12:22:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:153950K .......... .......... .......... .......... .......... 91% 102M 0s 2024-09-16T12:22:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:154000K .......... .......... .......... .......... .......... 91% 148M 0s 2024-09-16T12:22:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:154050K .......... .......... .......... .......... .......... 91% 140M 0s 2024-09-16T12:22:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:154100K .......... .......... .......... .......... .......... 91% 28.8M 0s 2024-09-16T12:22:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:154150K .......... .......... .......... .......... .......... 91% 51.8M 0s 2024-09-16T12:22:58.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:154200K .......... .......... .......... .......... .......... 91% 22.7M 0s 2024-09-16T12:22:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:154250K .......... .......... .......... .......... .......... 91% 37.1M 0s 2024-09-16T12:22:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:154300K .......... .......... .......... .......... .......... 91% 24.4M 0s 2024-09-16T12:22:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:154350K .......... .......... .......... .......... .......... 91% 23.4M 0s 2024-09-16T12:22:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:154400K .......... .......... .......... .......... .......... 91% 50.2M 0s 2024-09-16T12:22:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:154450K .......... .......... .......... .......... .......... 91% 24.6M 0s 2024-09-16T12:22:58.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:154500K .......... .......... .......... .......... .......... 91% 33.6M 0s 2024-09-16T12:22:58.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:154550K .......... .......... .......... .......... .......... 91% 24.8M 0s 2024-09-16T12:22:58.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:154600K .......... .......... .......... .......... .......... 91% 53.5M 0s 2024-09-16T12:22:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:154650K .......... .......... .......... .......... .......... 91% 23.7M 0s 2024-09-16T12:22:58.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:154700K .......... .......... .......... .......... .......... 91% 35.2M 0s 2024-09-16T12:22:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:154750K .......... .......... .......... .......... .......... 91% 17.6M 0s 2024-09-16T12:22:58.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:154800K .......... .......... .......... .......... .......... 91% 34.7M 0s 2024-09-16T12:22:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:154850K .......... .......... .......... .......... .......... 91% 26.7M 0s 2024-09-16T12:22:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:154900K .......... .......... .......... .......... .......... 91% 45.0M 0s 2024-09-16T12:22:58.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:154950K .......... .......... .......... .......... .......... 91% 23.7M 0s 2024-09-16T12:22:58.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:155000K .......... .......... .......... .......... .......... 91% 39.2M 0s 2024-09-16T12:22:58.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:155050K .......... .......... .......... .......... .......... 92% 24.0M 0s 2024-09-16T12:22:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:155100K .......... .......... .......... .......... .......... 92% 41.5M 0s 2024-09-16T12:22:59.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:155150K .......... .......... .......... .......... .......... 92% 28.1M 0s 2024-09-16T12:22:59.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:155200K .......... .......... .......... .......... .......... 92% 22.4M 0s 2024-09-16T12:22:59.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:155250K .......... .......... .......... .......... .......... 92% 26.6M 0s 2024-09-16T12:22:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:155300K .......... .......... .......... .......... .......... 92% 32.0M 0s 2024-09-16T12:22:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:155350K .......... .......... .......... .......... .......... 92% 27.7M 0s 2024-09-16T12:22:59.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:155400K .......... .......... .......... .......... .......... 92% 39.6M 0s 2024-09-16T12:22:59.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:155450K .......... .......... .......... .......... .......... 92% 25.3M 0s 2024-09-16T12:22:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:155500K .......... .......... .......... .......... .......... 92% 36.4M 0s 2024-09-16T12:22:59.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:155550K .......... .......... .......... .......... .......... 92% 22.9M 0s 2024-09-16T12:22:59.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:155600K .......... .......... .......... .......... .......... 92% 28.2M 0s 2024-09-16T12:22:59.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:155650K .......... .......... .......... .......... .......... 92% 31.4M 0s 2024-09-16T12:22:59.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:155700K .......... .......... .......... .......... .......... 92% 29.5M 0s 2024-09-16T12:22:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:155750K .......... .......... .......... .......... .......... 92% 33.3M 0s 2024-09-16T12:22:59.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:155800K .......... .......... .......... .......... .......... 92% 26.9M 0s 2024-09-16T12:22:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:155850K .......... .......... .......... .......... .......... 92% 39.9M 0s 2024-09-16T12:22:59.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:155900K .......... .......... .......... .......... .......... 92% 27.7M 0s 2024-09-16T12:22:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:155950K .......... .......... .......... .......... .......... 92% 24.3M 0s 2024-09-16T12:22:59.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:156000K .......... .......... .......... .......... .......... 92% 29.0M 0s 2024-09-16T12:22:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:156050K .......... .......... .......... .......... .......... 92% 27.1M 0s 2024-09-16T12:22:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:156100K .......... .......... .......... .......... .......... 92% 26.7M 0s 2024-09-16T12:22:59.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:156150K .......... .......... .......... .......... .......... 92% 38.6M 0s 2024-09-16T12:22:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:156200K .......... .......... .......... .......... .......... 92% 25.0M 0s 2024-09-16T12:22:59.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:156250K .......... .......... .......... .......... .......... 92% 25.3M 0s 2024-09-16T12:22:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:156300K .......... .......... .......... .......... .......... 92% 37.5M 0s 2024-09-16T12:22:59.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:156350K .......... .......... .......... .......... .......... 92% 23.8M 0s 2024-09-16T12:22:59.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:156400K .......... .......... .......... .......... .......... 92% 36.4M 0s 2024-09-16T12:22:59.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:156450K .......... .......... .......... .......... .......... 92% 29.6M 0s 2024-09-16T12:22:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:156500K .......... .......... .......... .......... .......... 92% 22.1M 0s 2024-09-16T12:22:59.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:156550K .......... .......... .......... .......... .......... 92% 35.9M 0s 2024-09-16T12:22:59.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:156600K .......... .......... .......... .......... .......... 92% 24.5M 0s 2024-09-16T12:22:59.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:156650K .......... .......... .......... .......... .......... 92% 45.1M 0s 2024-09-16T12:22:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:156700K .......... .......... .......... .......... .......... 92% 25.2M 0s 2024-09-16T12:22:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:156750K .......... .......... .......... .......... .......... 93% 24.2M 0s 2024-09-16T12:22:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:156800K .......... .......... .......... .......... .......... 93% 31.5M 0s 2024-09-16T12:22:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:156850K .......... .......... .......... .......... .......... 93% 29.0M 0s 2024-09-16T12:22:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:156900K .......... .......... .......... .......... .......... 93% 22.9M 0s 2024-09-16T12:22:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:156950K .......... .......... .......... .......... .......... 93% 41.4M 0s 2024-09-16T12:22:59.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:157000K .......... .......... .......... .......... .......... 93% 24.9M 0s 2024-09-16T12:22:59.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:157050K .......... .......... .......... .......... .......... 93% 37.2M 0s 2024-09-16T12:22:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:157100K .......... .......... .......... .......... .......... 93% 25.7M 0s 2024-09-16T12:22:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:157150K .......... .......... .......... .......... .......... 93% 25.0M 0s 2024-09-16T12:22:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:157200K .......... .......... .......... .......... .......... 93% 42.1M 0s 2024-09-16T12:22:59.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:157250K .......... .......... .......... .......... .......... 93% 25.4M 0s 2024-09-16T12:22:59.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:157300K .......... .......... .......... .......... .......... 93% 36.9M 0s 2024-09-16T12:22:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:157350K .......... .......... .......... .......... .......... 93% 24.7M 0s 2024-09-16T12:22:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:157400K .......... .......... .......... .......... .......... 93% 43.3M 0s 2024-09-16T12:22:59.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:157450K .......... .......... .......... .......... .......... 93% 25.2M 0s 2024-09-16T12:22:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:157500K .......... .......... .......... .......... .......... 93% 28.0M 0s 2024-09-16T12:22:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:157550K .......... .......... .......... .......... .......... 93% 21.2M 0s 2024-09-16T12:22:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:157600K .......... .......... .......... .......... .......... 93% 37.8M 0s 2024-09-16T12:22:59.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:157650K .......... .......... .......... .......... .......... 93% 23.9M 0s 2024-09-16T12:22:59.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:157700K .......... .......... .......... .......... .......... 93% 48.0M 0s 2024-09-16T12:22:59.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:157750K .......... .......... .......... .......... .......... 93% 23.8M 0s 2024-09-16T12:22:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:157800K .......... .......... .......... .......... .......... 93% 39.8M 0s 2024-09-16T12:22:59.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:157850K .......... .......... .......... .......... .......... 93% 20.1M 0s 2024-09-16T12:22:59.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:157900K .......... .......... .......... .......... .......... 93% 32.7M 0s 2024-09-16T12:22:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:157950K .......... .......... .......... .......... .......... 93% 23.3M 0s 2024-09-16T12:22:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:158000K .......... .......... .......... .......... .......... 93% 24.7M 0s 2024-09-16T12:22:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:158050K .......... .......... .......... .......... .......... 93% 33.1M 0s 2024-09-16T12:22:59.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:158100K .......... .......... .......... .......... .......... 93% 24.8M 0s 2024-09-16T12:22:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:158150K .......... .......... .......... .......... .......... 93% 24.9M 0s 2024-09-16T12:22:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:158200K .......... .......... .......... .......... .......... 93% 45.6M 0s 2024-09-16T12:22:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:158250K .......... .......... .......... .......... .......... 93% 35.6M 0s 2024-09-16T12:22:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:158300K .......... .......... .......... .......... .......... 93% 29.2M 0s 2024-09-16T12:22:59.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:158350K .......... .......... .......... .......... .......... 93% 19.1M 0s 2024-09-16T12:22:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:158400K .......... .......... .......... .......... .......... 93% 28.1M 0s 2024-09-16T12:22:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:158450K .......... .......... .......... .......... .......... 94% 47.7M 0s 2024-09-16T12:22:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:158500K .......... .......... .......... .......... .......... 94% 25.8M 0s 2024-09-16T12:22:59.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:158550K .......... .......... .......... .......... .......... 94% 34.9M 0s 2024-09-16T12:22:59.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:158600K .......... .......... .......... .......... .......... 94% 23.7M 0s 2024-09-16T12:22:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:158650K .......... .......... .......... .......... .......... 94% 25.5M 0s 2024-09-16T12:22:59.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:158700K .......... .......... .......... .......... .......... 94% 46.1M 0s 2024-09-16T12:22:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:158750K .......... .......... .......... .......... .......... 94% 23.8M 0s 2024-09-16T12:22:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:158800K .......... .......... .......... .......... .......... 94% 24.5M 0s 2024-09-16T12:22:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:158850K .......... .......... .......... .......... .......... 94% 38.9M 0s 2024-09-16T12:22:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:158900K .......... .......... .......... .......... .......... 94% 25.2M 0s 2024-09-16T12:22:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:158950K .......... .......... .......... .......... .......... 94% 41.4M 0s 2024-09-16T12:22:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:159000K .......... .......... .......... .......... .......... 94% 25.9M 0s 2024-09-16T12:22:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:159050K .......... .......... .......... .......... .......... 94% 34.9M 0s 2024-09-16T12:22:59.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:159100K .......... .......... .......... .......... .......... 94% 25.0M 0s 2024-09-16T12:22:59.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:159150K .......... .......... .......... .......... .......... 94% 23.0M 0s 2024-09-16T12:22:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:159200K .......... .......... .......... .......... .......... 94% 33.1M 0s 2024-09-16T12:22:59.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:159250K .......... .......... .......... .......... .......... 94% 36.8M 0s 2024-09-16T12:22:59.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:159300K .......... .......... .......... .......... .......... 94% 23.0M 0s 2024-09-16T12:22:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:159350K .......... .......... .......... .......... .......... 94% 24.9M 0s 2024-09-16T12:22:59.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:159400K .......... .......... .......... .......... .......... 94% 48.0M 0s 2024-09-16T12:22:59.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:159450K .......... .......... .......... .......... .......... 94% 32.8M 0s 2024-09-16T12:22:59.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:159500K .......... .......... .......... .......... .......... 94% 24.4M 0s 2024-09-16T12:22:59.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:159550K .......... .......... .......... .......... .......... 94% 24.8M 0s 2024-09-16T12:22:59.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:159600K .......... .......... .......... .......... .......... 94% 23.4M 0s 2024-09-16T12:22:59.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:159650K .......... .......... .......... .......... .......... 94% 75.3M 0s 2024-09-16T12:22:59.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:159700K .......... .......... .......... .......... .......... 94% 33.1M 0s 2024-09-16T12:22:59.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:159750K .......... .......... .......... .......... .......... 94% 23.1M 0s 2024-09-16T12:22:59.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:159800K .......... .......... .......... .......... .......... 94% 33.7M 0s 2024-09-16T12:22:59.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:159850K .......... .......... .......... .......... .......... 94% 24.3M 0s 2024-09-16T12:22:59.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:159900K .......... .......... .......... .......... .......... 94% 29.5M 0s 2024-09-16T12:22:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:159950K .......... .......... .......... .......... .......... 94% 25.1M 0s 2024-09-16T12:22:59.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:160000K .......... .......... .......... .......... .......... 94% 39.0M 0s 2024-09-16T12:22:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:160050K .......... .......... .......... .......... .......... 94% 22.5M 0s 2024-09-16T12:22:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:160100K .......... .......... .......... .......... .......... 95% 25.7M 0s 2024-09-16T12:22:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:160150K .......... .......... .......... .......... .......... 95% 34.4M 0s 2024-09-16T12:22:59.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:160200K .......... .......... .......... .......... .......... 95% 44.7M 0s 2024-09-16T12:22:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:160250K .......... .......... .......... .......... .......... 95% 25.0M 0s 2024-09-16T12:22:59.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:160300K .......... .......... .......... .......... .......... 95% 24.9M 0s 2024-09-16T12:22:59.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:160350K .......... .......... .......... .......... .......... 95% 22.8M 0s 2024-09-16T12:22:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:160400K .......... .......... .......... .......... .......... 95% 46.4M 0s 2024-09-16T12:22:59.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:160450K .......... .......... .......... .......... .......... 95% 26.0M 0s 2024-09-16T12:22:59.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:160500K .......... .......... .......... .......... .......... 95% 36.4M 0s 2024-09-16T12:22:59.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:160550K .......... .......... .......... .......... .......... 95% 24.2M 0s 2024-09-16T12:22:59.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:160600K .......... .......... .......... .......... .......... 95% 37.7M 0s 2024-09-16T12:22:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:160650K .......... .......... .......... .......... .......... 95% 23.7M 0s 2024-09-16T12:22:59.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:160700K .......... .......... .......... .......... .......... 95% 49.2M 0s 2024-09-16T12:22:59.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:160750K .......... .......... .......... .......... .......... 95% 23.4M 0s 2024-09-16T12:22:59.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:160800K .......... .......... .......... .......... .......... 95% 24.4M 0s 2024-09-16T12:22:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:160850K .......... .......... .......... .......... .......... 95% 37.9M 0s 2024-09-16T12:22:59.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:160900K .......... .......... .......... .......... .......... 95% 24.4M 0s 2024-09-16T12:22:59.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:160950K .......... .......... .......... .......... .......... 95% 24.9M 0s 2024-09-16T12:22:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:161000K .......... .......... .......... .......... .......... 95% 40.6M 0s 2024-09-16T12:22:59.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:161050K .......... .......... .......... .......... .......... 95% 25.0M 0s 2024-09-16T12:22:59.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:161100K .......... .......... .......... .......... .......... 95% 36.8M 0s 2024-09-16T12:22:59.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:161150K .......... .......... .......... .......... .......... 95% 29.1M 0s 2024-09-16T12:22:59.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:161200K .......... .......... .......... .......... .......... 95% 23.0M 0s 2024-09-16T12:22:59.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:161250K .......... .......... .......... .......... .......... 95% 42.5M 0s 2024-09-16T12:22:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:161300K .......... .......... .......... .......... .......... 95% 24.0M 0s 2024-09-16T12:22:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:161350K .......... .......... .......... .......... .......... 95% 36.8M 0s 2024-09-16T12:22:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:161400K .......... .......... .......... .......... .......... 95% 23.8M 0s 2024-09-16T12:22:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:161450K .......... .......... .......... .......... .......... 95% 23.5M 0s 2024-09-16T12:22:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:161500K .......... .......... .......... .......... .......... 95% 53.9M 0s 2024-09-16T12:22:59.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:161550K .......... .......... .......... .......... .......... 95% 23.7M 0s 2024-09-16T12:22:59.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:161600K .......... .......... .......... .......... .......... 95% 23.6M 0s 2024-09-16T12:22:59.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:161650K .......... .......... .......... .......... .......... 95% 40.3M 0s 2024-09-16T12:22:59.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:161700K .......... .......... .......... .......... .......... 95% 23.9M 0s 2024-09-16T12:22:59.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:161750K .......... .......... .......... .......... .......... 95% 39.9M 0s 2024-09-16T12:22:59.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:161800K .......... .......... .......... .......... .......... 96% 35.0M 0s 2024-09-16T12:22:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:161850K .......... .......... .......... .......... .......... 96% 27.1M 0s 2024-09-16T12:22:59.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:161900K .......... .......... .......... .......... .......... 96% 19.1M 0s 2024-09-16T12:22:59.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:161950K .......... .......... .......... .......... .......... 96% 26.0M 0s 2024-09-16T12:22:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:162000K .......... .......... .......... .......... .......... 96% 24.8M 0s 2024-09-16T12:22:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:162050K .......... .......... .......... .......... .......... 96% 43.4M 0s 2024-09-16T12:22:59.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:162100K .......... .......... .......... .......... .......... 96% 26.5M 0s 2024-09-16T12:22:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:162150K .......... .......... .......... .......... .......... 96% 31.7M 0s 2024-09-16T12:22:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:162200K .......... .......... .......... .......... .......... 96% 23.6M 0s 2024-09-16T12:22:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:162250K .......... .......... .......... .......... .......... 96% 53.1M 0s 2024-09-16T12:22:59.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:162300K .......... .......... .......... .......... .......... 96% 23.6M 0s 2024-09-16T12:22:59.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:162350K .......... .......... .......... .......... .......... 96% 24.0M 0s 2024-09-16T12:22:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:162400K .......... .......... .......... .......... .......... 96% 36.9M 0s 2024-09-16T12:22:59.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:162450K .......... .......... .......... .......... .......... 96% 25.2M 0s 2024-09-16T12:22:59.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:162500K .......... .......... .......... .......... .......... 96% 45.6M 0s 2024-09-16T12:22:59.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:162550K .......... .......... .......... .......... .......... 96% 24.2M 0s 2024-09-16T12:22:59.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:162600K .......... .......... .......... .......... .......... 96% 33.7M 0s 2024-09-16T12:22:59.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:162650K .......... .......... .......... .......... .......... 96% 25.1M 0s 2024-09-16T12:22:59.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:162700K .......... .......... .......... .......... .......... 96% 24.7M 0s 2024-09-16T12:22:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:162750K .......... .......... .......... .......... .......... 96% 25.6M 0s 2024-09-16T12:22:59.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:162800K .......... .......... .......... .......... .......... 96% 47.5M 0s 2024-09-16T12:22:59.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:162850K .......... .......... .......... .......... .......... 96% 23.6M 0s 2024-09-16T12:22:59.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:162900K .......... .......... .......... .......... .......... 96% 35.0M 0s 2024-09-16T12:22:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:162950K .......... .......... .......... .......... .......... 96% 25.9M 0s 2024-09-16T12:22:59.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:163000K .......... .......... .......... .......... .......... 96% 44.2M 0s 2024-09-16T12:22:59.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:163050K .......... .......... .......... .......... .......... 96% 24.1M 0s 2024-09-16T12:22:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:163100K .......... .......... .......... .......... .......... 96% 37.5M 0s 2024-09-16T12:22:59.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:163150K .......... .......... .......... .......... .......... 96% 22.4M 0s 2024-09-16T12:22:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:163200K .......... .......... .......... .......... .......... 96% 25.7M 0s 2024-09-16T12:22:59.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:163250K .......... .......... .......... .......... .......... 96% 24.0M 0s 2024-09-16T12:22:59.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:163300K .......... .......... .......... .......... .......... 96% 49.1M 0s 2024-09-16T12:22:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:163350K .......... .......... .......... .......... .......... 96% 23.7M 0s 2024-09-16T12:22:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:163400K .......... .......... .......... .......... .......... 96% 48.0M 0s 2024-09-16T12:22:59.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:163450K .......... .......... .......... .......... .......... 96% 22.0M 0s 2024-09-16T12:22:59.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:163500K .......... .......... .......... .......... .......... 97% 46.6M 0s 2024-09-16T12:22:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:163550K .......... .......... .......... .......... .......... 97% 24.6M 0s 2024-09-16T12:22:59.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:163600K .......... .......... .......... .......... .......... 97% 23.9M 0s 2024-09-16T12:22:59.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:163650K .......... .......... .......... .......... .......... 97% 37.3M 0s 2024-09-16T12:22:59.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:163700K .......... .......... .......... .......... .......... 97% 23.2M 0s 2024-09-16T12:22:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:163750K .......... .......... .......... .......... .......... 97% 24.1M 0s 2024-09-16T12:22:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:163800K .......... .......... .......... .......... .......... 97% 57.3M 0s 2024-09-16T12:22:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:163850K .......... .......... .......... .......... .......... 97% 23.0M 0s 2024-09-16T12:22:59.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:163900K .......... .......... .......... .......... .......... 97% 38.6M 0s 2024-09-16T12:22:59.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:163950K .......... .......... .......... .......... .......... 97% 23.6M 0s 2024-09-16T12:22:59.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:164000K .......... .......... .......... .......... .......... 97% 24.6M 0s 2024-09-16T12:22:59.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:164050K .......... .......... .......... .......... .......... 97% 47.6M 0s 2024-09-16T12:22:59.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:164100K .......... .......... .......... .......... .......... 97% 25.6M 0s 2024-09-16T12:22:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:164150K .......... .......... .......... .......... .......... 97% 36.0M 0s 2024-09-16T12:22:59.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:164200K .......... .......... .......... .......... .......... 97% 23.9M 0s 2024-09-16T12:22:59.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:164250K .......... .......... .......... .......... .......... 97% 22.6M 0s 2024-09-16T12:22:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:164300K .......... .......... .......... .......... .......... 97% 54.0M 0s 2024-09-16T12:22:59.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:164350K .......... .......... .......... .......... .......... 97% 25.0M 0s 2024-09-16T12:22:59.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:164400K .......... .......... .......... .......... .......... 97% 31.6M 0s 2024-09-16T12:22:59.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:164450K .......... .......... .......... .......... .......... 97% 25.2M 0s 2024-09-16T12:22:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:164500K .......... .......... .......... .......... .......... 97% 25.3M 0s 2024-09-16T12:22:59.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:164550K .......... .......... .......... .......... .......... 97% 48.6M 0s 2024-09-16T12:22:59.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:164600K .......... .......... .......... .......... .......... 97% 34.7M 0s 2024-09-16T12:22:59.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:164650K .......... .......... .......... .......... .......... 97% 26.0M 0s 2024-09-16T12:22:59.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:164700K .......... .......... .......... .......... .......... 97% 24.6M 0s 2024-09-16T12:22:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:164750K .......... .......... .......... .......... .......... 97% 23.1M 0s 2024-09-16T12:22:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:164800K .......... .......... .......... .......... .......... 97% 24.0M 0s 2024-09-16T12:22:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:164850K .......... .......... .......... .......... .......... 97% 54.7M 0s 2024-09-16T12:22:59.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:164900K .......... .......... .......... .......... .......... 97% 21.7M 0s 2024-09-16T12:22:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:164950K .......... .......... .......... .......... .......... 97% 36.6M 0s 2024-09-16T12:22:59.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:165000K .......... .......... .......... .......... .......... 97% 37.2M 0s 2024-09-16T12:22:59.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:165050K .......... .......... .......... .......... .......... 97% 32.5M 0s 2024-09-16T12:22:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:165100K .......... .......... .......... .......... .......... 97% 22.4M 0s 2024-09-16T12:22:59.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:165150K .......... .......... .......... .......... .......... 98% 24.7M 0s 2024-09-16T12:22:59.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:165200K .......... .......... .......... .......... .......... 98% 36.5M 0s 2024-09-16T12:22:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:165250K .......... .......... .......... .......... .......... 98% 25.5M 0s 2024-09-16T12:22:59.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:165300K .......... .......... .......... .......... .......... 98% 44.1M 0s 2024-09-16T12:22:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:165350K .......... .......... .......... .......... .......... 98% 24.3M 0s 2024-09-16T12:22:59.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:165400K .......... .......... .......... .......... .......... 98% 35.2M 0s 2024-09-16T12:22:59.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:165450K .......... .......... .......... .......... .......... 98% 25.1M 0s 2024-09-16T12:22:59.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:165500K .......... .......... .......... .......... .......... 98% 23.5M 0s 2024-09-16T12:22:59.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:165550K .......... .......... .......... .......... .......... 98% 28.3M 0s 2024-09-16T12:22:59.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:165600K .......... .......... .......... .......... .......... 98% 36.5M 0s 2024-09-16T12:22:59.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:165650K .......... .......... .......... .......... .......... 98% 31.5M 0s 2024-09-16T12:22:59.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:165700K .......... .......... .......... .......... .......... 98% 28.1M 0s 2024-09-16T12:22:59.360 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:165750K .......... .......... .......... .......... .......... 98% 24.1M 0s 2024-09-16T12:22:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:165800K .......... .......... .......... .......... .......... 98% 45.9M 0s 2024-09-16T12:22:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:165850K .......... .......... .......... .......... .......... 98% 24.6M 0s 2024-09-16T12:22:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:165900K .......... .......... .......... .......... .......... 98% 38.1M 0s 2024-09-16T12:22:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:165950K .......... .......... .......... .......... .......... 98% 24.3M 0s 2024-09-16T12:22:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:166000K .......... .......... .......... .......... .......... 98% 23.3M 0s 2024-09-16T12:22:59.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:166050K .......... .......... .......... .......... .......... 98% 25.2M 0s 2024-09-16T12:22:59.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:166100K .......... .......... .......... .......... .......... 98% 41.1M 0s 2024-09-16T12:22:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:166150K .......... .......... .......... .......... .......... 98% 24.2M 0s 2024-09-16T12:22:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:166200K .......... .......... .......... .......... .......... 98% 38.6M 0s 2024-09-16T12:22:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:166250K .......... .......... .......... .......... .......... 98% 22.7M 0s 2024-09-16T12:22:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:166300K .......... .......... .......... .......... .......... 98% 46.0M 0s 2024-09-16T12:22:59.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:166350K .......... .......... .......... .......... .......... 98% 24.8M 0s 2024-09-16T12:22:59.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:166400K .......... .......... .......... .......... .......... 98% 24.4M 0s 2024-09-16T12:22:59.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:166450K .......... .......... .......... .......... .......... 98% 39.6M 0s 2024-09-16T12:22:59.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:166500K .......... .......... .......... .......... .......... 98% 25.1M 0s 2024-09-16T12:22:59.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:166550K .......... .......... .......... .......... .......... 98% 22.0M 0s 2024-09-16T12:22:59.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:166600K .......... .......... .......... .......... .......... 98% 49.1M 0s 2024-09-16T12:22:59.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:166650K .......... .......... .......... .......... .......... 98% 24.6M 0s 2024-09-16T12:22:59.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:166700K .......... .......... .......... .......... .......... 98% 37.8M 0s 2024-09-16T12:22:59.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:166750K .......... .......... .......... .......... .......... 98% 23.5M 0s 2024-09-16T12:22:59.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:166800K .......... .......... .......... .......... .......... 98% 26.2M 0s 2024-09-16T12:22:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:166850K .......... .......... .......... .......... .......... 99% 49.1M 0s 2024-09-16T12:22:59.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:166900K .......... .......... .......... .......... .......... 99% 32.5M 0s 2024-09-16T12:22:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:166950K .......... .......... .......... .......... .......... 99% 25.4M 0s 2024-09-16T12:22:59.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:167000K .......... .......... .......... .......... .......... 99% 24.6M 0s 2024-09-16T12:22:59.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:167050K .......... .......... .......... .......... .......... 99% 47.7M 0s 2024-09-16T12:22:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:167100K .......... .......... .......... .......... .......... 99% 27.7M 0s 2024-09-16T12:22:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:167150K .......... .......... .......... .......... .......... 99% 20.8M 0s 2024-09-16T12:22:59.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:167200K .......... .......... .......... .......... .......... 99% 29.3M 0s 2024-09-16T12:22:59.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:167250K .......... .......... .......... .......... .......... 99% 30.8M 0s 2024-09-16T12:22:59.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:167300K .......... .......... .......... .......... .......... 99% 24.3M 0s 2024-09-16T12:22:59.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:167350K .......... .......... .......... .......... .......... 99% 46.1M 0s 2024-09-16T12:22:59.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:167400K .......... .......... .......... .......... .......... 99% 24.2M 0s 2024-09-16T12:22:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:167450K .......... .......... .......... .......... .......... 99% 38.7M 0s 2024-09-16T12:22:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:167500K .......... .......... .......... .......... .......... 99% 25.5M 0s 2024-09-16T12:22:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:167550K .......... .......... .......... .......... .......... 99% 24.0M 0s 2024-09-16T12:22:59.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:167600K .......... .......... .......... .......... .......... 99% 48.2M 0s 2024-09-16T12:22:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:167650K .......... .......... .......... .......... .......... 99% 23.0M 0s 2024-09-16T12:22:59.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:167700K .......... .......... .......... .......... .......... 99% 36.0M 0s 2024-09-16T12:22:59.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:167750K .......... .......... .......... .......... .......... 99% 19.9M 0s 2024-09-16T12:22:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:167800K .......... .......... .......... .......... .......... 99% 31.8M 0s 2024-09-16T12:22:59.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:167850K .......... .......... .......... .......... .......... 99% 65.1M 0s 2024-09-16T12:22:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:167900K .......... .......... .......... .......... .......... 99% 20.9M 0s 2024-09-16T12:22:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:167950K .......... .......... .......... .......... .......... 99% 26.5M 0s 2024-09-16T12:22:59.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:168000K .......... .......... .......... .......... .......... 99% 32.0M 0s 2024-09-16T12:22:59.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:168050K .......... .......... .......... .......... .......... 99% 24.7M 0s 2024-09-16T12:22:59.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:168100K .......... .......... .......... .......... .......... 99% 44.4M 0s 2024-09-16T12:22:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:168150K .......... .......... .......... .......... .......... 99% 24.0M 0s 2024-09-16T12:22:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:168200K .......... .......... .......... .......... .......... 99% 41.5M 0s 2024-09-16T12:22:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:168250K .......... .......... .......... .......... .......... 99% 23.0M 0s 2024-09-16T12:22:59.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:168300K .......... .......... .......... .......... .......... 99% 27.2M 0s 2024-09-16T12:22:59.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:168350K .......... .......... .......... .......... .......... 99% 21.4M 0s 2024-09-16T12:22:59.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:168400K .......... .......... .......... .......... .......... 99% 47.1M 0s 2024-09-16T12:22:59.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:168450K .......... .......... .......... .......... .......... 99% 28.6M 0s 2024-09-16T12:22:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:168500K .......... .......... .......... .......... .......... 99% 31.4M 0s 2024-09-16T12:22:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:168550K .......... .......... . 100% 104M=5.8s 2024-09-16T12:22:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2024-09-16T12:22:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:2024-09-16 12:22:59 (28.5 MB/s) - 'linux-5.4.tar.gz' saved [172616875/172616875] 2024-09-16T12:22:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: 2024-09-16T12:22:59.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..10} 2024-09-16T12:22:59.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS=() 2024-09-16T12:22:59.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS=() 2024-09-16T12:22:59.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS=() 2024-09-16T12:22:59.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS=() 2024-09-16T12:22:59.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ RET=0 2024-09-16T12:22:59.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:22:59.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary1 2024-09-16T12:22:59.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary1 2024-09-16T12:22:59.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-16T12:22:59.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:22:59.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:22:59.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:22:59.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:22:59.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:22:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:22:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:22:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:22:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:22:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-16T12:22:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:22:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:22:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:22:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:22:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:22:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:22:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:22:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:22:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:22:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:22:59.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-16T12:22:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:22:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-16T12:22:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:22:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:22:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:22:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:22:59.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-16T12:23:00.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:23:00.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-16T12:23:00.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary1': 2024-09-16T12:23:00.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:23:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:23:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:23:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 10ad9135f695 2024-09-16T12:23:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.10ad9135f695 2024-09-16T12:23:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:23:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:23:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:23:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:23:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:22:59 2024 2024-09-16T12:23:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:22:59 2024 2024-09-16T12:23:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:22:59 2024 2024-09-16T12:23:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:23:00.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:23:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: eb4f75b8-2cf0-409a-b490-25d6035748ff 2024-09-16T12:23:00.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:23:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:23:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary1 2024-09-16T12:23:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd0 2024-09-16T12:23:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:23:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd0 2024-09-16T12:23:01.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:23:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:23:01.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:23:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 87001373-cc50-4eb0-98bd-0a3bf6f9bae3 2024-09-16T12:23:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:23:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:23:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:23:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:23:01.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:23:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:23:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:23:01.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:23:01.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary1 2024-09-16T12:23:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd0 test-primary1 2024-09-16T12:23:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:23:01.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary1 2024-09-16T12:23:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:23:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-16T12:23:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:23:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:23:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:23:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary2 2024-09-16T12:23:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary2 2024-09-16T12:23:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:01.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-16T12:23:01.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:23:01.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary1 2024-09-16T12:23:01.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:01.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:01.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:23:01.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:01.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:01.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:01.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:01.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:23:01.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:23:01.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary1 2024-09-16T12:23:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:23:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-16T12:23:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:23:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-16T12:23:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:23:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:01.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:23:01.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:23:01.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:23:01.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:01.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-16T12:23:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:23:01.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-16T12:23:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:23:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:23:01.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-16T12:23:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:23:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-16T12:23:02.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5 2024-09-16T12:23:02.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:23:02.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:02.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-16T12:23:02.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary2': 2024-09-16T12:23:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:23:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:23:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:23:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 10bd60259723 2024-09-16T12:23:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.10bd60259723 2024-09-16T12:23:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:23:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:23:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:23:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:23:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:23:01 2024 2024-09-16T12:23:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:23:01 2024 2024-09-16T12:23:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:23:01 2024 2024-09-16T12:23:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:23:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:23:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: d0aa1661-aaad-47bc-a934-2a5891ec1d70 2024-09-16T12:23:02.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:23:02.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:23:02.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary2 2024-09-16T12:23:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd1 2024-09-16T12:23:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:23:02.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd1 2024-09-16T12:23:02.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:23:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:23:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:23:02.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: bd368504-9385-45ab-80b9-bd4a08e601f0 2024-09-16T12:23:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:23:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:23:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:23:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:23:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:23:03.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:23:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:23:03.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:23:03.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary2 2024-09-16T12:23:03.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd1 test-primary2 2024-09-16T12:23:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:23:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary2 2024-09-16T12:23:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:23:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-16T12:23:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:23:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:23:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:23:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:23:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary3 2024-09-16T12:23:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary3 2024-09-16T12:23:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:23:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary2 2024-09-16T12:23:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-16T12:23:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:23:03.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary2 2024-09-16T12:23:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-16T12:23:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:23:03.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:23:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:23:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:03.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-16T12:23:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:23:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:23:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:23:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:23:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:03.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-16T12:23:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:23:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-16T12:23:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:23:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:23:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-16T12:23:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:23:03.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-16T12:23:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 7 2024-09-16T12:23:03.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:23:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:03.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-16T12:23:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary3': 2024-09-16T12:23:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:23:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:23:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:23:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 10cd345e262d 2024-09-16T12:23:03.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.10cd345e262d 2024-09-16T12:23:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:23:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:23:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:23:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:23:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:23:03 2024 2024-09-16T12:23:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:23:03 2024 2024-09-16T12:23:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:23:03 2024 2024-09-16T12:23:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:23:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:23:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: cc1f33be-5579-4d01-8ea8-2efb5bab2552 2024-09-16T12:23:03.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:23:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:23:03.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary3 2024-09-16T12:23:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd2 2024-09-16T12:23:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:23:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd2 2024-09-16T12:23:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:23:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:23:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:23:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: dfab3fc9-1ad0-4b87-8df8-447edf7bdf61 2024-09-16T12:23:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:23:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:23:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:23:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:23:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:23:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:23:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:23:04.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:23:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary3 2024-09-16T12:23:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd2 test-primary3 2024-09-16T12:23:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:23:04.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary3 2024-09-16T12:23:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:23:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-16T12:23:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:23:04.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:23:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:23:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:23:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary4 2024-09-16T12:23:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary4 2024-09-16T12:23:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:23:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-16T12:23:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:23:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:04.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:23:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:23:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:23:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-16T12:23:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:23:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:23:04.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:23:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:23:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-16T12:23:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary3 2024-09-16T12:23:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary3 2024-09-16T12:23:04.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-16T12:23:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:23:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-16T12:23:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:23:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:23:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-16T12:23:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:23:04.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-16T12:23:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:23:04.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 9 2024-09-16T12:23:04.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:04.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-16T12:23:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary4': 2024-09-16T12:23:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:23:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:23:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:23:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 10da3ab30e5b 2024-09-16T12:23:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.10da3ab30e5b 2024-09-16T12:23:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:23:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:23:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:23:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:23:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:23:04 2024 2024-09-16T12:23:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:23:04 2024 2024-09-16T12:23:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:23:04 2024 2024-09-16T12:23:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:23:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:23:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: a83eb5d8-a522-4c70-9ed3-260b39b46e23 2024-09-16T12:23:04.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:23:04.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:23:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary4 2024-09-16T12:23:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd3 2024-09-16T12:23:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:23:04.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd3 2024-09-16T12:23:04.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:23:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:23:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:23:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 2bae9699-3896-48b6-8ad1-510ab5c52c1b 2024-09-16T12:23:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:23:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:23:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:23:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:23:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:23:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:23:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:23:05.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:23:05.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary4 2024-09-16T12:23:05.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd3 test-primary4 2024-09-16T12:23:05.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:23:05.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary4 2024-09-16T12:23:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:23:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-16T12:23:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:23:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:23:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:23:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary4 2024-09-16T12:23:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:23:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary5 2024-09-16T12:23:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary5 2024-09-16T12:23:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:05.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:23:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-16T12:23:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:23:05.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary4 2024-09-16T12:23:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-16T12:23:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:23:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:23:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:23:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-16T12:23:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:23:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:23:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:23:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:23:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-16T12:23:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:23:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-16T12:23:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:23:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:23:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-16T12:23:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:23:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-16T12:23:05.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:05.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:23:05.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:05.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:05.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:23:05.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:23:05.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 11 2024-09-16T12:23:05.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:23:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-16T12:23:05.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary5': 2024-09-16T12:23:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:23:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:23:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:23:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 10ede212bd5b 2024-09-16T12:23:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.10ede212bd5b 2024-09-16T12:23:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:23:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:23:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:23:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:23:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:23:05 2024 2024-09-16T12:23:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:23:05 2024 2024-09-16T12:23:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:23:05 2024 2024-09-16T12:23:05.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:23:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:23:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 3a299e6e-0be4-4a52-9286-05028efdde9c 2024-09-16T12:23:05.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:23:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:23:05.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary5 2024-09-16T12:23:05.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd4 2024-09-16T12:23:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:23:05.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd4 2024-09-16T12:23:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:23:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:23:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:23:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: bee5a6eb-7aee-4e2d-b9f3-135ca7501baf 2024-09-16T12:23:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:23:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:23:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:23:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:23:05.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:23:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:23:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:23:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:23:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary5 2024-09-16T12:23:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd4 test-primary5 2024-09-16T12:23:06.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:23:06.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary5 2024-09-16T12:23:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:23:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-16T12:23:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:23:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:23:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:23:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:23:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:23:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:23:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary6 2024-09-16T12:23:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary6 2024-09-16T12:23:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-16T12:23:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:23:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:23:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:23:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-16T12:23:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:23:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:23:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:23:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:23:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-16T12:23:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary5 2024-09-16T12:23:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary5 2024-09-16T12:23:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-16T12:23:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:23:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-16T12:23:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:23:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:23:06.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-16T12:23:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:23:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-16T12:23:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:06.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:23:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:23:06.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:23:06.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:23:06.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 14 2024-09-16T12:23:06.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-16T12:23:06.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary6': 2024-09-16T12:23:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:23:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:23:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:23:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 10fec3de3f2b 2024-09-16T12:23:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.10fec3de3f2b 2024-09-16T12:23:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:23:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:23:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:23:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:23:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:23:06 2024 2024-09-16T12:23:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:23:06 2024 2024-09-16T12:23:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:23:06 2024 2024-09-16T12:23:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:23:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:23:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 56e0c85b-bcd6-45e9-baf8-ab981b2ca159 2024-09-16T12:23:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:23:06.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:23:06.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary6 2024-09-16T12:23:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd5 2024-09-16T12:23:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:23:06.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd5 2024-09-16T12:23:06.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:23:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:23:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:23:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: a9c79099-e295-444c-9250-a0194a218f7b 2024-09-16T12:23:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:23:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:23:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:23:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:23:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:23:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:23:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:23:07.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:23:07.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary6 2024-09-16T12:23:07.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd5 test-primary6 2024-09-16T12:23:07.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:23:07.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary6 2024-09-16T12:23:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:23:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-16T12:23:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:23:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:23:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:23:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:23:07.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:23:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:23:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary7 2024-09-16T12:23:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary7 2024-09-16T12:23:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-16T12:23:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:23:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:23:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:23:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-16T12:23:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:23:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:23:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:23:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:23:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary6 2024-09-16T12:23:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-16T12:23:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary6 2024-09-16T12:23:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-16T12:23:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:23:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-16T12:23:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:23:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:23:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-16T12:23:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:23:07.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-16T12:23:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:23:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:23:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:23:07.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 13 2024-09-16T12:23:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 18 2024-09-16T12:23:07.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:23:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-16T12:23:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary7': 2024-09-16T12:23:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:23:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:23:07.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:23:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 11121ea690ff 2024-09-16T12:23:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.11121ea690ff 2024-09-16T12:23:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:23:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:23:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:23:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:23:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:23:07 2024 2024-09-16T12:23:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:23:07 2024 2024-09-16T12:23:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:23:07 2024 2024-09-16T12:23:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:23:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:23:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: b9c65b6f-b177-4f69-b32f-158e1d24fdcc 2024-09-16T12:23:07.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:23:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:23:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary7 2024-09-16T12:23:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd6 2024-09-16T12:23:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:23:07.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd6 2024-09-16T12:23:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:23:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:23:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:23:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: f234029b-1bc4-4a90-8832-d4a29ff848a2 2024-09-16T12:23:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:23:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:23:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:23:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:23:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:23:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:23:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:23:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:23:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary7 2024-09-16T12:23:08.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd6 test-primary7 2024-09-16T12:23:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:23:08.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary7 2024-09-16T12:23:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:23:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:23:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:23:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary8 2024-09-16T12:23:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary8 2024-09-16T12:23:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-16T12:23:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:23:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:23:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:23:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-16T12:23:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:23:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:23:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:23:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:23:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-16T12:23:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-16T12:23:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary7 2024-09-16T12:23:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary7 2024-09-16T12:23:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-16T12:23:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:23:08.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:23:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:23:08.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:23:08.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:23:08.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-16T12:23:08.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:08.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:08.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:23:08.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:23:08.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-16T12:23:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:23:08.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-16T12:23:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:23:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:23:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:23:08.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 17 2024-09-16T12:23:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 22 2024-09-16T12:23:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:23:08.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-16T12:23:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary8': 2024-09-16T12:23:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:23:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:23:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:23:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1124e12bd93c 2024-09-16T12:23:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1124e12bd93c 2024-09-16T12:23:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:23:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:23:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:23:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:23:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:23:08 2024 2024-09-16T12:23:08.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:23:08 2024 2024-09-16T12:23:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:23:08 2024 2024-09-16T12:23:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:23:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:23:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 9eb4b9fc-5335-4349-afe2-ff0112468825 2024-09-16T12:23:08.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:23:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:23:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary8 2024-09-16T12:23:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd7 2024-09-16T12:23:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:23:08.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd7 2024-09-16T12:23:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:23:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:23:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:23:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 4b7e5bbf-4406-4fd0-8d91-68a671b7fa07 2024-09-16T12:23:08.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:23:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:23:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:23:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:23:08.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:23:09.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:23:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:23:09.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:23:09.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary8 2024-09-16T12:23:09.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd7 test-primary8 2024-09-16T12:23:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:23:09.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary8 2024-09-16T12:23:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:23:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:23:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:23:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary9 2024-09-16T12:23:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary9 2024-09-16T12:23:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-16T12:23:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:23:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:23:09.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:23:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-16T12:23:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:23:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:23:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:23:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:23:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-16T12:23:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary8 2024-09-16T12:23:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary8 2024-09-16T12:23:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-16T12:23:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-16T12:23:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:23:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:23:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:23:09.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:23:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:23:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-16T12:23:09.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:09.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:09.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:23:09.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:23:09.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-16T12:23:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:23:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-16T12:23:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:23:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:23:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:23:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 21 2024-09-16T12:23:09.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:23:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 26 2024-09-16T12:23:09.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:09.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-16T12:23:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary9': 2024-09-16T12:23:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:23:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:23:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:23:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1137185de3ed 2024-09-16T12:23:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1137185de3ed 2024-09-16T12:23:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:23:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:23:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:23:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:23:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:23:09 2024 2024-09-16T12:23:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:23:09 2024 2024-09-16T12:23:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:23:09 2024 2024-09-16T12:23:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:23:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:23:09.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 72aa8251-8228-4686-a7e8-cfad99ad25c1 2024-09-16T12:23:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:23:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:23:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary9 2024-09-16T12:23:09.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd8 2024-09-16T12:23:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:23:09.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd8 2024-09-16T12:23:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:23:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:23:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:23:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: ff1ffced-18a7-4ab5-8463-704b6e6d8c39 2024-09-16T12:23:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:23:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:23:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:23:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:23:09.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:23:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:23:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:23:10.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:23:10.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary9 2024-09-16T12:23:10.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd8 test-primary9 2024-09-16T12:23:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:23:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary9 2024-09-16T12:23:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:23:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:23:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:23:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary10 2024-09-16T12:23:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary10 2024-09-16T12:23:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-16T12:23:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:23:10.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:23:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:23:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-16T12:23:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:23:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:23:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:23:10.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:23:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:23:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-16T12:23:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-16T12:23:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:23:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:10.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:23:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:23:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:23:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary9 2024-09-16T12:23:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary9 2024-09-16T12:23:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-16T12:23:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:23:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-16T12:23:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:23:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:23:10.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-16T12:23:10.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:23:10.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-16T12:23:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:23:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:23:10.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:23:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:23:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:23:10.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:23:10.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 25 2024-09-16T12:23:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:10.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 30 2024-09-16T12:23:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:23:10.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:10.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-16T12:23:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary10': 2024-09-16T12:23:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:23:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:23:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:23:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 114a1712754a 2024-09-16T12:23:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.114a1712754a 2024-09-16T12:23:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:23:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:23:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:23:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:23:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:23:10 2024 2024-09-16T12:23:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:23:10 2024 2024-09-16T12:23:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:23:10 2024 2024-09-16T12:23:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:23:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:23:10.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 76ea3943-f961-4a54-a62d-d75534b59757 2024-09-16T12:23:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:23:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:23:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary10 2024-09-16T12:23:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd9 2024-09-16T12:23:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:23:10.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd9 2024-09-16T12:23:10.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:23:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:23:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:23:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 72c22f3d-7505-4d94-8c1d-13ba839776a9 2024-09-16T12:23:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:23:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:23:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:23:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:23:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:23:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:23:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:23:11.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:23:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary10 2024-09-16T12:23:11.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd9 test-primary10 2024-09-16T12:23:11.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:23:11.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary10 2024-09-16T12:23:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:23:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:23:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:23:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary10 2024-09-16T12:23:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 27445 2024-09-16T12:23:11.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary10 2024-09-16T12:23:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-16T12:23:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-16T12:23:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:23:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:23:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:23:11.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:23:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:23:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-16T12:23:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:23:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:23:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:23:11.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 29 2024-09-16T12:23:11.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 35 2024-09-16T12:23:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:23:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:23:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:23:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:23:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:23:12.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:23:12.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 34 2024-09-16T12:23:12.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:23:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:23:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:23:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 33 2024-09-16T12:23:12.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:23:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:23:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:23:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 39 2024-09-16T12:23:13.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 38 2024-09-16T12:23:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:13.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:23:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:23:13.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:23:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:14.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:23:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:23:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:23:14.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 43 2024-09-16T12:23:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 42 2024-09-16T12:23:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:23:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:23:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:23:14.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:23:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:15.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:15.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:23:15.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:23:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:23:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:23:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:23:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 47 2024-09-16T12:23:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 46 2024-09-16T12:23:15.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:15.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:23:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:23:16.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:23:16.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 51 2024-09-16T12:23:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:23:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:23:16.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:23:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 50 2024-09-16T12:23:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:16.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:17.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 55 2024-09-16T12:23:17.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 54 2024-09-16T12:23:17.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:23:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:17.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:23:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:23:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:23:17.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:23:17.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:23:18.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 58 2024-09-16T12:23:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 59 2024-09-16T12:23:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:18.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:23:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:23:18.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:23:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:23:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:23:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:23:19.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 62 2024-09-16T12:23:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:19.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 63 2024-09-16T12:23:19.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:19.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:23:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:23:19.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:23:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:19.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:23:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:23:19.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:23:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:23:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:23:20.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:23:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 67 2024-09-16T12:23:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:23:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:23:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:23:20.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:20.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 66 2024-09-16T12:23:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:20.821 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:23:20.816+0000 7f74def64700 -1 reset not still connected to 0x559441c01790 2024-09-16T12:23:20.821 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:23:20.816+0000 7f74def64700 -1 reset not still connected to 0x559441c01d40 2024-09-16T12:23:20.821 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:23:20.816+0000 7f74def64700 -1 reset not still connected to 0x55944e3fa820 2024-09-16T12:23:20.821 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:23:20.816+0000 7f74def64700 -1 reset not still connected to 0x559452bd6a90 2024-09-16T12:23:20.822 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:23:20.816+0000 7f74def64700 -1 reset not still connected to 0x559454618b60 2024-09-16T12:23:21.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 70 2024-09-16T12:23:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:23:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:23:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:23:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:23:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:23:21.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:23:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:21.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 71 2024-09-16T12:23:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:23:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:23:22.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:23:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:23:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:23:22.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:23:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 74 2024-09-16T12:23:22.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 75 2024-09-16T12:23:22.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:22.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:23.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 78 2024-09-16T12:23:23.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:23.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:23:23.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:23.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:23.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:23:23.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:23:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:23.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 79 2024-09-16T12:23:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:23:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:23:23.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:23:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:23:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:24.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:23:24.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:23:24.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 82 2024-09-16T12:23:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 83 2024-09-16T12:23:24.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:23:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:23:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:23:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:23:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:23:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:23:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:23:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:23:25.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:23:25.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 87 2024-09-16T12:23:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 86 2024-09-16T12:23:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:26.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:23:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:23:26.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:23:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 90 2024-09-16T12:23:26.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:23:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:23:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:23:26.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:26.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 91 2024-09-16T12:23:26.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:27.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 95 2024-09-16T12:23:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 94 2024-09-16T12:23:27.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:27.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:23:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:23:27.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:23:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:23:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:23:27.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:23:28.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 98 2024-09-16T12:23:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 99 2024-09-16T12:23:28.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:28.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:23:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:23:28.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:23:28.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:23:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:23:28.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:23:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 103 2024-09-16T12:23:29.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 102 2024-09-16T12:23:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:23:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:29.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:23:29.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:23:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:23:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:23:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:23:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:23:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:23:30.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:23:30.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:30.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:23:30.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:30.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:30.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:23:30.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:23:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 106 2024-09-16T12:23:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 107 2024-09-16T12:23:30.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:31.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 110 2024-09-16T12:23:31.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 111 2024-09-16T12:23:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:23:31.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:23:31.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:23:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:23:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:23:31.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:23:31.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:31.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:32.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 113 2024-09-16T12:23:32.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 112 2024-09-16T12:23:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:23:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:23:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:23:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:23:32.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:23:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:23:32.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 118 2024-09-16T12:23:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 119 2024-09-16T12:23:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:33.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:23:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:23:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:23:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:23:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:33.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:23:33.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:23:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:23:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:23:34.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:23:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:23:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:23:34.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:23:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 122 2024-09-16T12:23:34.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 123 2024-09-16T12:23:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:23:35.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:23:35.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:23:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:23:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:23:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:23:35.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 127 2024-09-16T12:23:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 126 2024-09-16T12:23:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:35.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:23:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:36.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:23:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:23:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:23:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:23:36.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:23:36.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 130 2024-09-16T12:23:36.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 131 2024-09-16T12:23:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 135 2024-09-16T12:23:37.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 134 2024-09-16T12:23:37.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:37.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:23:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:23:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:23:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:23:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:23:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:23:38.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 139 2024-09-16T12:23:38.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 138 2024-09-16T12:23:38.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:38.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:23:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:23:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:23:38.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:23:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:23:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:23:39.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 143 2024-09-16T12:23:39.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 142 2024-09-16T12:23:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:23:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:23:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:23:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:23:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:23:39.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:23:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:23:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:23:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:23:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:23:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:23:40.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:23:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 146 2024-09-16T12:23:40.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 147 2024-09-16T12:23:40.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:23:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:23:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:23:41.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 151 2024-09-16T12:23:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:23:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:23:41.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:23:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 150 2024-09-16T12:23:41.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:41.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:42.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:23:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:23:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:23:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:23:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:23:42.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:23:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 155 2024-09-16T12:23:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 154 2024-09-16T12:23:42.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:23:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:23:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:23:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:23:43.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:23:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:23:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:44.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:23:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:23:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:23:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:23:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:23:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:23:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 159 2024-09-16T12:23:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 160 2024-09-16T12:23:44.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 158 2024-09-16T12:23:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 161 2024-09-16T12:23:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:23:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:23:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:23:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:23:45.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:45.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:45.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:23:45.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:23:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 166 2024-09-16T12:23:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 167 2024-09-16T12:23:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 170 2024-09-16T12:23:46.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 168 2024-09-16T12:23:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 174 2024-09-16T12:23:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 175 2024-09-16T12:23:47.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:23:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:23:47.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:23:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:23:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:23:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:23:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:23:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:23:47.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:23:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:23:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:23:47.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:23:47.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:23:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:23:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:23:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:23:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:23:48.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:23:49.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 181 2024-09-16T12:23:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 178 2024-09-16T12:23:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:23:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:23:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:23:49.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:23:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:23:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:23:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 179 2024-09-16T12:23:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 180 2024-09-16T12:23:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:49.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:49.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:50.154 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:23:50.148+0000 7f74def64700 -1 reset not still connected to 0x559441c06f70 2024-09-16T12:23:50.155 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:23:50.148+0000 7f74def64700 -1 reset not still connected to 0x55944e3fa820 2024-09-16T12:23:50.155 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:23:50.148+0000 7f74def64700 -1 reset not still connected to 0x559452bd6a90 2024-09-16T12:23:50.155 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:23:50.148+0000 7f74def64700 -1 reset not still connected to 0x559454618b60 2024-09-16T12:23:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:23:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:23:50.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:23:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 186 2024-09-16T12:23:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 187 2024-09-16T12:23:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:50.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:23:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:23:50.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:23:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:50.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:51.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 189 2024-09-16T12:23:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 191 2024-09-16T12:23:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:52.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 192 2024-09-16T12:23:52.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 195 2024-09-16T12:23:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:23:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:23:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:23:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:23:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:23:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:23:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:23:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:23:52.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:23:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:23:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:23:52.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:23:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:53.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:23:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:23:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:23:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:23:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:23:53.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:23:54.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:23:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:23:54.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:23:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:23:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:23:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:23:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 201 2024-09-16T12:23:54.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 200 2024-09-16T12:23:54.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 199 2024-09-16T12:23:54.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 198 2024-09-16T12:23:54.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:54.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:54.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:54.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:55.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:23:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:23:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:23:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:23:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:23:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:23:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 206 2024-09-16T12:23:55.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 207 2024-09-16T12:23:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 208 2024-09-16T12:23:56.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 211 2024-09-16T12:23:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:57.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 214 2024-09-16T12:23:57.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 215 2024-09-16T12:23:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:57.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:23:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:23:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:23:57.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:57.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:23:57.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:57.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:57.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:23:57.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:23:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:23:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:23:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:23:57.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:57.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:23:57.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:57.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:57.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:23:57.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:23:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:23:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:23:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:23:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:23:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:58.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:23:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:23:59.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 219 2024-09-16T12:23:59.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 218 2024-09-16T12:23:59.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 220 2024-09-16T12:23:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 221 2024-09-16T12:23:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:23:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:23:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:23:59.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:59.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:59.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:23:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:23:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:23:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:23:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:23:59.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:23:59.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:23:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:00.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:24:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:24:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:24:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:24:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:24:00.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:24:00.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 226 2024-09-16T12:24:00.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 227 2024-09-16T12:24:00.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:01.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 231 2024-09-16T12:24:01.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 230 2024-09-16T12:24:01.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:01.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 232 2024-09-16T12:24:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:24:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:24:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:24:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:02.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:24:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:24:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:24:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:24:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:24:02.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:24:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 233 2024-09-16T12:24:02.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:24:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:24:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:24:02.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:24:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:24:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:24:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:24:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:24:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:24:03.737 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:24:03.731+0000 7ff4f6946700 -1 reset not still connected to 0x557540963ad0 2024-09-16T12:24:03.737 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:24:03.731+0000 7ff4f6946700 -1 reset not still connected to 0x55754fed2f70 2024-09-16T12:24:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:24:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:24:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:24:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:24:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:24:04.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:24:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 239 2024-09-16T12:24:04.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 240 2024-09-16T12:24:04.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 238 2024-09-16T12:24:04.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:04.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 241 2024-09-16T12:24:04.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:04.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:24:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:24:05.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:24:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:24:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:24:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:24:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 247 2024-09-16T12:24:05.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 246 2024-09-16T12:24:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:05.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 248 2024-09-16T12:24:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 250 2024-09-16T12:24:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:07.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 255 2024-09-16T12:24:07.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 254 2024-09-16T12:24:07.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:07.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:07.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:24:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:24:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:24:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:24:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:24:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:24:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:24:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:24:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:24:07.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:24:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:24:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:24:08.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:24:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:24:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:24:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:08.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:24:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:24:08.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:24:09.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 259 2024-09-16T12:24:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:24:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:24:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:24:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:09.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:24:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:24:09.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:24:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:09.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 258 2024-09-16T12:24:09.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 260 2024-09-16T12:24:09.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 261 2024-09-16T12:24:09.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:09.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:09.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:10.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:10.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:24:10.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:24:10.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 267 2024-09-16T12:24:10.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 266 2024-09-16T12:24:10.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:10.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:11.218 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:24:11.211+0000 7f61b1855700 -1 reset not still connected to 0x557b22af11e0 2024-09-16T12:24:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 271 2024-09-16T12:24:11.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 273 2024-09-16T12:24:11.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 272 2024-09-16T12:24:11.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 270 2024-09-16T12:24:11.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:11.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:12.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:24:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:24:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:24:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:24:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:24:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:24:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:24:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:24:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:24:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:24:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:24:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:24:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:24:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:24:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:24:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:24:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:24:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:24:14.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 278 2024-09-16T12:24:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:24:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:24:14.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:24:14.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 280 2024-09-16T12:24:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 281 2024-09-16T12:24:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:24:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:24:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:24:14.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 279 2024-09-16T12:24:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:24:14.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:24:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:24:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:24:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:24:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:24:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:14.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:14.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 287 2024-09-16T12:24:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 286 2024-09-16T12:24:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 292 2024-09-16T12:24:16.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 293 2024-09-16T12:24:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 291 2024-09-16T12:24:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:16.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 290 2024-09-16T12:24:16.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:24:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:24:17.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:24:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:24:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:24:17.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:24:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:24:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:24:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:24:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:24:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:24:17.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:24:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:24:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:24:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:24:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:24:19.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 299 2024-09-16T12:24:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 301 2024-09-16T12:24:19.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:19.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:24:19.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:19.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:19.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:24:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:24:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 300 2024-09-16T12:24:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:24:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:24:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:24:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 298 2024-09-16T12:24:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:24:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:24:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:24:19.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:24:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:24:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:24:20.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 307 2024-09-16T12:24:20.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 306 2024-09-16T12:24:20.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:20.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 313 2024-09-16T12:24:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 312 2024-09-16T12:24:21.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 311 2024-09-16T12:24:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 310 2024-09-16T12:24:21.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:21.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:24:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:24:22.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:24:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:24:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:24:22.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:24:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:24:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:24:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:24:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:24:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:24:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:24:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:24:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:24:23.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:24:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:24:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:24:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:24:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:24:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:24:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:24:24.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:24:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:24:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:24:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:24:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:24:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:24:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 319 2024-09-16T12:24:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:24:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:24:24.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:24:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 321 2024-09-16T12:24:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 320 2024-09-16T12:24:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 318 2024-09-16T12:24:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:25.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 327 2024-09-16T12:24:25.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 326 2024-09-16T12:24:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 328 2024-09-16T12:24:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 329 2024-09-16T12:24:26.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 330 2024-09-16T12:24:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 333 2024-09-16T12:24:26.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:27.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:27.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:27.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:27.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:24:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:24:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:24:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:24:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:24:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:24:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:24:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:24:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:24:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:24:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:24:27.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:24:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:24:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:24:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:24:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:24:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:24:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:24:29.133 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:24:29.126+0000 7f61b1855700 -1 reset not still connected to 0x557b22af11e0 2024-09-16T12:24:29.133 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:24:29.130+0000 7f61b1855700 -1 reset not still connected to 0x557b2f60ad00 2024-09-16T12:24:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 340 2024-09-16T12:24:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 339 2024-09-16T12:24:29.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 338 2024-09-16T12:24:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:24:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:24:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:24:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 341 2024-09-16T12:24:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:24:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:24:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:24:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:24:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:24:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:24:29.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:29.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:24:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:24:29.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:24:30.194 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T12:24:30.187+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c299bd40 2024-09-16T12:24:30.195 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T12:24:30.187+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c8c76ea0 2024-09-16T12:24:30.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 347 2024-09-16T12:24:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 346 2024-09-16T12:24:30.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:31.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 351 2024-09-16T12:24:31.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 350 2024-09-16T12:24:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 353 2024-09-16T12:24:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 352 2024-09-16T12:24:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:31.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:24:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:24:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:24:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:24:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:24:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:24:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:32.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:24:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:24:32.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:24:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:24:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:24:32.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:24:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:24:33.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:24:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:24:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:24:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:24:33.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:24:34.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 358 2024-09-16T12:24:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 361 2024-09-16T12:24:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:24:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:24:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:24:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 359 2024-09-16T12:24:34.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:24:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:24:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:24:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:24:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:24:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:24:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:24:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:24:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:24:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:34.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 360 2024-09-16T12:24:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 367 2024-09-16T12:24:35.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 366 2024-09-16T12:24:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:36.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 371 2024-09-16T12:24:36.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 373 2024-09-16T12:24:36.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 372 2024-09-16T12:24:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 370 2024-09-16T12:24:36.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:36.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:24:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:24:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:24:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:24:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:24:37.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:24:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:24:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:24:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:24:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:24:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:24:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:24:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:24:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:24:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:24:38.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:24:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:24:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:24:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:39.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:24:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:24:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:24:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:24:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:24:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:24:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 379 2024-09-16T12:24:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:24:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:24:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:24:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:24:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:39.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:39.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:24:39.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:24:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 380 2024-09-16T12:24:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 378 2024-09-16T12:24:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 381 2024-09-16T12:24:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:39.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 387 2024-09-16T12:24:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 386 2024-09-16T12:24:40.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 389 2024-09-16T12:24:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 390 2024-09-16T12:24:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 392 2024-09-16T12:24:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 393 2024-09-16T12:24:41.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:41.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:41.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:24:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:24:42.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:24:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:24:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:24:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:24:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:24:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:24:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:24:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:24:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:24:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:24:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:24:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:24:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:24:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:24:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:24:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:24:44.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:44.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:24:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:24:44.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:24:44.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 399 2024-09-16T12:24:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 398 2024-09-16T12:24:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 401 2024-09-16T12:24:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:24:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:24:44.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:24:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:24:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:24:44.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:24:44.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:44.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 400 2024-09-16T12:24:44.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:24:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:24:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:24:44.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:44.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 406 2024-09-16T12:24:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 407 2024-09-16T12:24:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 410 2024-09-16T12:24:46.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 412 2024-09-16T12:24:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 413 2024-09-16T12:24:46.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 411 2024-09-16T12:24:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:46.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:46.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:47.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:47.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:24:47.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:47.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:47.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:24:47.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:24:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:24:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:47.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:24:47.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:24:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:24:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:24:47.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:24:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:47.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:24:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:24:47.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:24:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:24:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:24:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:24:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:24:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:24:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:24:49.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 419 2024-09-16T12:24:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 420 2024-09-16T12:24:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:49.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:24:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:24:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:24:49.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 421 2024-09-16T12:24:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 418 2024-09-16T12:24:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:24:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:24:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:24:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:24:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:24:49.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:24:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:24:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:49.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:24:49.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:24:49.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:50.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 427 2024-09-16T12:24:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 426 2024-09-16T12:24:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:51.129 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:24:51.126+0000 7f4071c2f700 -1 reset not still connected to 0x563591bfa000 2024-09-16T12:24:51.129 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:24:51.126+0000 7f4071c2f700 -1 reset not still connected to 0x563591d7dad0 2024-09-16T12:24:51.131 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:24:51.126+0000 7f4071c2f700 -1 reset not still connected to 0x563592528820 2024-09-16T12:24:51.131 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:24:51.126+0000 7f4071c2f700 -1 reset not still connected to 0x563592528ea0 2024-09-16T12:24:51.131 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:24:51.126+0000 7f4071c2f700 -1 reset not still connected to 0x5635925295f0 2024-09-16T12:24:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 432 2024-09-16T12:24:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 431 2024-09-16T12:24:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:51.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 430 2024-09-16T12:24:51.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 433 2024-09-16T12:24:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:24:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:24:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:24:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:24:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:24:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:24:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:24:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:24:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:24:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:24:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:24:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:24:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:24:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:24:54.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:24:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:24:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:24:54.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:24:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:24:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:24:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:24:54.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:24:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:24:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:24:54.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 440 2024-09-16T12:24:55.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:24:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:24:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:24:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:24:55.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:24:55.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:24:55.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 438 2024-09-16T12:24:55.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 439 2024-09-16T12:24:55.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 441 2024-09-16T12:24:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:55.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:56.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 447 2024-09-16T12:24:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 446 2024-09-16T12:24:56.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:56.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 453 2024-09-16T12:24:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 451 2024-09-16T12:24:57.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 450 2024-09-16T12:24:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 452 2024-09-16T12:24:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:57.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:57.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:24:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:24:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:24:58.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:24:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:24:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:24:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:24:58.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:24:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:24:58.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:24:58.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:58.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:24:58.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:58.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:58.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:24:58.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:24:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:24:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:59.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:59.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:24:59.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:24:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:24:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:24:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:24:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:24:59.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:24:59.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:25:00.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 460 2024-09-16T12:25:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 461 2024-09-16T12:25:00.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 459 2024-09-16T12:25:00.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:25:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:25:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:25:00.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 458 2024-09-16T12:25:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:25:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:25:00.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:25:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:00.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:25:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:25:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:25:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:25:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:25:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:25:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:01.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 467 2024-09-16T12:25:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 466 2024-09-16T12:25:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:01.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:02.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 473 2024-09-16T12:25:02.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 472 2024-09-16T12:25:02.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 470 2024-09-16T12:25:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:02.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 471 2024-09-16T12:25:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:02.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:03.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:25:03.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:03.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:03.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:25:03.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:25:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:25:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:25:03.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:25:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:25:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:25:03.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:25:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:25:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:25:03.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:25:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:25:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:25:04.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:25:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:25:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:25:04.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:25:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:25:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:05.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:25:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:25:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:25:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:25:05.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:25:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:25:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:25:05.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:25:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 481 2024-09-16T12:25:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 479 2024-09-16T12:25:05.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 480 2024-09-16T12:25:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:25:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:25:05.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:25:05.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 478 2024-09-16T12:25:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:05.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:06.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 487 2024-09-16T12:25:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 486 2024-09-16T12:25:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:07.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 491 2024-09-16T12:25:07.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:07.069 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:25:07.061+0000 7ff4f6946700 -1 reset not still connected to 0x557541654d00 2024-09-16T12:25:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 490 2024-09-16T12:25:07.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 492 2024-09-16T12:25:07.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 493 2024-09-16T12:25:07.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:08.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:08.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:25:08.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:25:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:25:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:25:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:25:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:25:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:25:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:25:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:25:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:25:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:25:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:25:09.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:25:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:25:09.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:25:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:09.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:25:09.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:09.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:09.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:25:09.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:25:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 501 2024-09-16T12:25:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:25:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:25:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:25:10.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 498 2024-09-16T12:25:10.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 499 2024-09-16T12:25:10.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:25:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:25:10.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:25:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 500 2024-09-16T12:25:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:25:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:10.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:10.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:25:10.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:25:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:25:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:25:10.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:25:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 507 2024-09-16T12:25:11.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 506 2024-09-16T12:25:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:12.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 510 2024-09-16T12:25:12.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 511 2024-09-16T12:25:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:12.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:12.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 512 2024-09-16T12:25:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 513 2024-09-16T12:25:12.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:12.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:25:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:25:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:25:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:25:13.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:25:13.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:25:13.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:25:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:25:13.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:25:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:25:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:13.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:25:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:25:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:25:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:14.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:25:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:25:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:25:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:25:14.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:25:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:25:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:25:15.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:25:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:25:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:25:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:25:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 520 2024-09-16T12:25:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 521 2024-09-16T12:25:15.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 519 2024-09-16T12:25:15.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:25:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:25:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:25:15.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 518 2024-09-16T12:25:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:25:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:25:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:25:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:15.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:16.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 527 2024-09-16T12:25:16.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 526 2024-09-16T12:25:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:16.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:17.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 531 2024-09-16T12:25:17.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 533 2024-09-16T12:25:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 530 2024-09-16T12:25:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:17.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:17.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 532 2024-09-16T12:25:17.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:25:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:25:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:25:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:25:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:25:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:25:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:25:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:25:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:25:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:25:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:25:18.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:25:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:25:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:25:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:25:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:25:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:25:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:25:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 540 2024-09-16T12:25:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:25:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:25:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:25:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 538 2024-09-16T12:25:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 539 2024-09-16T12:25:20.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:25:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:25:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:25:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:25:20.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:25:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:25:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:20.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:20.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:25:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:25:20.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:25:20.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 541 2024-09-16T12:25:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:21.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 547 2024-09-16T12:25:21.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 546 2024-09-16T12:25:21.126 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:25:21.117+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0ea0 2024-09-16T12:25:21.126 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:25:21.121+0000 7f61b1855700 -1 reset not still connected to 0x557b22af11e0 2024-09-16T12:25:21.126 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:25:21.121+0000 7f61b1855700 -1 reset not still connected to 0x557b2f60ad00 2024-09-16T12:25:21.126 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:25:21.121+0000 7f61b1855700 -1 reset not still connected to 0x557b3179e0d0 2024-09-16T12:25:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:22.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 550 2024-09-16T12:25:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 551 2024-09-16T12:25:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 552 2024-09-16T12:25:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 553 2024-09-16T12:25:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:22.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:22.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:25:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:25:23.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:25:23.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:25:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:25:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:25:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:25:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:25:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:25:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:25:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:25:23.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:25:23.692 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:25:23.686+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e36ab1e0 2024-09-16T12:25:23.692 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:25:23.686+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e5d9d2b0 2024-09-16T12:25:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:25:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:25:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:25:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:25:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:25:24.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:25:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:25:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:25:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:25:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 559 2024-09-16T12:25:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:25:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:25:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:25:25.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 558 2024-09-16T12:25:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 560 2024-09-16T12:25:25.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:25:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:25:25.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:25:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:25:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:25:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:25:25.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 561 2024-09-16T12:25:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:26.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 566 2024-09-16T12:25:26.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 567 2024-09-16T12:25:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:27.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 571 2024-09-16T12:25:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 573 2024-09-16T12:25:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 570 2024-09-16T12:25:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:27.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:27.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:27.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 572 2024-09-16T12:25:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:28.096 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:25:28.090+0000 7f74def64700 -1 reset not still connected to 0x559441c00a90 2024-09-16T12:25:28.097 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:25:28.090+0000 7f74def64700 -1 reset not still connected to 0x559441c01d40 2024-09-16T12:25:28.097 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:25:28.090+0000 7f74def64700 -1 reset not still connected to 0x559441c06f70 2024-09-16T12:25:28.097 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:25:28.090+0000 7f74def64700 -1 reset not still connected to 0x559441c07ba0 2024-09-16T12:25:28.097 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:25:28.090+0000 7f74def64700 -1 reset not still connected to 0x559452bd6a90 2024-09-16T12:25:28.097 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:25:28.090+0000 7f74def64700 -1 reset not still connected to 0x559454618b60 2024-09-16T12:25:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:25:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:25:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:25:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:28.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:25:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:25:28.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:25:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:25:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:25:28.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:25:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:25:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:25:28.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:25:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:25:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:25:29.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:25:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:25:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:25:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:25:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 579 2024-09-16T12:25:30.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 578 2024-09-16T12:25:30.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:25:30.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 27588 2024-09-16T12:25:30.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 580 2024-09-16T12:25:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:25:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:25:30.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:25:30.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:25:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:25:30.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:25:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 581 2024-09-16T12:25:30.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:25:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:30.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:25:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:25:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:31.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 587 2024-09-16T12:25:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 586 2024-09-16T12:25:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:31.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:32.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 591 2024-09-16T12:25:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 592 2024-09-16T12:25:32.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 590 2024-09-16T12:25:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:32.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:32.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:25:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:25:33.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:25:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:25:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:25:33.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:25:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:25:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 27725 2024-09-16T12:25:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:25:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 27861 2024-09-16T12:25:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:25:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:25:34.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:25:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:25:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 28017 2024-09-16T12:25:35.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 596 2024-09-16T12:25:35.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:35.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 597 2024-09-16T12:25:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:25:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:25:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:25:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:25:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:25:35.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:25:35.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:25:35.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 28172 2024-09-16T12:25:35.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:25:35.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 28329 2024-09-16T12:25:36.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 600 2024-09-16T12:25:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 602 2024-09-16T12:25:37.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:25:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:25:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 28485 2024-09-16T12:25:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:25:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 28642 2024-09-16T12:25:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:25:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 28818 2024-09-16T12:25:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T12:25:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:25:40.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 27446 2024-09-16T12:25:53.670 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:25:53.664+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T12:28:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:28:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:28:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:28:01.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 27589 2024-09-16T12:28:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:28:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:28:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:28:03.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 27726 2024-09-16T12:28:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:28:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:28:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:28:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 27862 2024-09-16T12:28:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:28:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:28:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:28:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 28018 2024-09-16T12:28:06.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:28:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:28:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:28:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 28173 2024-09-16T12:28:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:28:07.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:28:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:28:07.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 28330 2024-09-16T12:28:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:28:08.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:28:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:28:08.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 28486 2024-09-16T12:28:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:28:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:28:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:28:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 28643 2024-09-16T12:28:10.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:28:10.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:28:10.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:28:10.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 28819 2024-09-16T12:28:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:28:11.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:28:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T12:28:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:28:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:28:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:28:11.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 1 2024-09-16T12:28:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:28:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd0 2024-09-16T12:28:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:28:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 2 2024-09-16T12:28:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary1 2024-09-16T12:28:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary1 2024-09-16T12:28:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:28:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary1 2024-09-16T12:28:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd1 2024-09-16T12:28:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary2 2024-09-16T12:28:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary2 2024-09-16T12:28:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:28:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary2 2024-09-16T12:28:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:28:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:28:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 3 2024-09-16T12:28:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:28:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:28:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 4 2024-09-16T12:28:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd2 2024-09-16T12:28:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary3 2024-09-16T12:28:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary3 2024-09-16T12:28:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:28:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary3 2024-09-16T12:28:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:28:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:28:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd3 2024-09-16T12:28:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary4 2024-09-16T12:28:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary4 2024-09-16T12:28:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:28:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary4 2024-09-16T12:28:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:28:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:28:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 6 2024-09-16T12:28:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 5 2024-09-16T12:28:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:28:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:28:11.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd5 2024-09-16T12:28:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary6 2024-09-16T12:28:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary6 2024-09-16T12:28:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:28:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd4 2024-09-16T12:28:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 7 2024-09-16T12:28:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary6 2024-09-16T12:28:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary5 2024-09-16T12:28:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary5 2024-09-16T12:28:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:28:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary5 2024-09-16T12:28:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:28:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:28:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd6 2024-09-16T12:28:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary7 2024-09-16T12:28:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary7 2024-09-16T12:28:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:28:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary7 2024-09-16T12:28:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 8 2024-09-16T12:28:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:28:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:28:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd7 2024-09-16T12:28:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary8 2024-09-16T12:28:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary8 2024-09-16T12:28:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:28:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary8 2024-09-16T12:28:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 9 2024-09-16T12:28:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:28:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:28:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 34846 2024-09-16T12:28:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd8 2024-09-16T12:28:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary9 2024-09-16T12:28:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 10 2024-09-16T12:28:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary9 2024-09-16T12:28:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:28:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd9 2024-09-16T12:28:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary10 2024-09-16T12:28:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary10 2024-09-16T12:28:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:28:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary10 2024-09-16T12:28:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary9 2024-09-16T12:28:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd7 2024-09-16T12:28:11.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:28:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:28:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd1 2024-09-16T12:28:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:28:11.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd9 2024-09-16T12:28:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:28:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd3 2024-09-16T12:28:11.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd5 2024-09-16T12:28:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:28:11.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd8 2024-09-16T12:28:11.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:28:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:28:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd6 2024-09-16T12:28:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:28:11.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd4 2024-09-16T12:28:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd0 2024-09-16T12:28:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:28:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:28:11.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd2 2024-09-16T12:29:07.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=b256cce4b466352497b9199e4b4b07ab 2024-09-16T12:29:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary10 2024-09-16T12:29:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=60bb4de8cd16f3161de4094727c7c900 2024-09-16T12:29:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary2 2024-09-16T12:29:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=e57968b3040835636ac9d172b59254bc 2024-09-16T12:29:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary6 2024-09-16T12:29:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=f3fe2949ca96e9fe275baa290be06617 2024-09-16T12:29:08.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary7 2024-09-16T12:29:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=3d7f357ec0731645da61d9d75ecc5bbf 2024-09-16T12:29:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary5 2024-09-16T12:29:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=b5b5283b5350f653cbd2285551ca497f 2024-09-16T12:29:08.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary4 2024-09-16T12:29:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=52bfc6783792e40af528c03b5a47f6eb 2024-09-16T12:29:08.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary3 2024-09-16T12:29:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=88b027a455ec7975b66ccab54701a7a2 2024-09-16T12:29:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary8 2024-09-16T12:29:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=80adcf0245b6cf843d097e456b523d5d 2024-09-16T12:29:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary1 2024-09-16T12:29:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=e427e1c4c7314e6400c4561097a65c40 2024-09-16T12:29:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary9 2024-09-16T12:29:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-16T12:29:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:29:08.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-16T12:29:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-16T12:29:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:29:09.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-16T12:29:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-16T12:29:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:29:09.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-16T12:29:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-16T12:29:09.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:09.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:09.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:29:09.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-16T12:29:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-16T12:29:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:29:09.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-16T12:29:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-16T12:29:09.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:29:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-16T12:29:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-16T12:29:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:29:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-16T12:29:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-16T12:29:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:29:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-16T12:29:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-16T12:29:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:29:09.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-16T12:29:09.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-16T12:29:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:29:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-16T12:29:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:29:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:29:10.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:29:10.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:29:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:29:10.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:29:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-16T12:29:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:29:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:29:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:29:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-16T12:29:10.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:29:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:29:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-16T12:29:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:29:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:29:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-16T12:29:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:29:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:29:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-16T12:29:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:10.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:29:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:29:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-16T12:29:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:29:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:29:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-16T12:29:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:29:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:10.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:29:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:29:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:29:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-16T12:29:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:29:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:29:10.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-16T12:29:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:29:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:29:11.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:29:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-16T12:29:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:29:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:11.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:11.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:29:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-16T12:29:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-16T12:29:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:29:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:29:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T12:29:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:11.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T12:29:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-16T12:29:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:29:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:11.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:11.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:11.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T12:29:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-16T12:29:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:29:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:11.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-16T12:29:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:29:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:11.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:11.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T12:29:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-16T12:29:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-16T12:29:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:29:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:29:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T12:29:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:11.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T12:29:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:11.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T12:29:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T12:29:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 9eb4b9fc-5335-4349-afe2-ff0112468825 2024-09-16T12:29:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:29:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:29:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:08 2024-09-16T12:29:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:29:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:29:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2146304.0,"last_snapshot_bytes":2531328,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489534,"remote_snapshot_timestamp":1726489534,"replay_state":"idle"} 2024-09-16T12:29:11.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:28:53' 2024-09-16T12:29:11.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:11.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T12:29:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 76ea3943-f961-4a54-a62d-d75534b59757 2024-09-16T12:29:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:29:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:29:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:08 2024-09-16T12:29:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:29:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:29:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2140160.0,"last_snapshot_bytes":2609152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489536,"remote_snapshot_timestamp":1726489536,"replay_state":"idle"} 2024-09-16T12:29:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:28:53' 2024-09-16T12:29:11.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T12:29:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 56e0c85b-bcd6-45e9-baf8-ab981b2ca159 2024-09-16T12:29:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:29:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:29:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:11.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:08 2024-09-16T12:29:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:29:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:29:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2764800.0,"last_snapshot_bytes":1757184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489531,"remote_snapshot_timestamp":1726489531,"replay_state":"idle"} 2024-09-16T12:29:11.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:28:53' 2024-09-16T12:29:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T12:29:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 3a299e6e-0be4-4a52-9286-05028efdde9c 2024-09-16T12:29:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:29:11.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:29:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:08 2024-09-16T12:29:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:29:11.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:29:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2160640.0,"last_snapshot_bytes":2560000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489531,"remote_snapshot_timestamp":1726489531,"replay_state":"idle"} 2024-09-16T12:29:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:28:53' 2024-09-16T12:29:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T12:29:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a83eb5d8-a522-4c70-9ed3-260b39b46e23 2024-09-16T12:29:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:29:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:29:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:11.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:08 2024-09-16T12:29:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:29:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:29:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2148352.0,"last_snapshot_bytes":2560000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489530,"remote_snapshot_timestamp":1726489530,"replay_state":"idle"} 2024-09-16T12:29:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:28:53' 2024-09-16T12:29:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:11.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T12:29:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b9c65b6f-b177-4f69-b32f-158e1d24fdcc 2024-09-16T12:29:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:29:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:29:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:08 2024-09-16T12:29:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:11.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:29:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:29:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3192832.0,"last_snapshot_bytes":4702208,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489534,"remote_snapshot_timestamp":1726489534,"replay_state":"idle"} 2024-09-16T12:29:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:28:53' 2024-09-16T12:29:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T12:29:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: cc1f33be-5579-4d01-8ea8-2efb5bab2552 2024-09-16T12:29:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:29:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:29:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:11.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:08 2024-09-16T12:29:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:29:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:29:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2138112.0,"last_snapshot_bytes":2613248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489529,"remote_snapshot_timestamp":1726489529,"replay_state":"idle"} 2024-09-16T12:29:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:28:53' 2024-09-16T12:29:11.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:29:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:29:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:29:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:29:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:29:11.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:29:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:29:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:29:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:29:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:29:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:29:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:29:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:29:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:29:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-16T12:29:11.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:29:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:11.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T12:29:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-16T12:29:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:29:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:11.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:11.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T12:29:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T12:29:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: eb4f75b8-2cf0-409a-b490-25d6035748ff 2024-09-16T12:29:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:29:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:29:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:08 2024-09-16T12:29:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:29:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:29:12.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:12' 2024-09-16T12:29:12.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:29:12.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:12.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:29:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T12:29:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: d0aa1661-aaad-47bc-a934-2a5891ec1d70 2024-09-16T12:29:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:29:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:29:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:08 2024-09-16T12:29:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:29:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:29:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:12' 2024-09-16T12:29:12.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:29:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:12.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:29:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-16T12:29:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:29:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:12.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T12:29:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T12:29:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 72aa8251-8228-4686-a7e8-cfad99ad25c1 2024-09-16T12:29:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:29:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:29:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:08 2024-09-16T12:29:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:29:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:29:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2160640.0,"last_snapshot_bytes":1712128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489535,"remote_snapshot_timestamp":1726489535,"replay_state":"idle"} 2024-09-16T12:29:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:28:53' 2024-09-16T12:29:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:29:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:29:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-16T12:29:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:29:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-16T12:29:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-16T12:29:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-16T12:29:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-16T12:29:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:29:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:29:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:29:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-16T12:29:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:29:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:13.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-16T12:29:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:29:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:29:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T12:29:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T12:29:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T12:29:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T12:29:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T12:29:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T12:29:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:13.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T12:29:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T12:29:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 76ea3943-f961-4a54-a62d-d75534b59757 2024-09-16T12:29:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13 2024-09-16T12:29:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:29:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:13.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13' 2024-09-16T12:29:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T12:29:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 9eb4b9fc-5335-4349-afe2-ff0112468825 2024-09-16T12:29:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13 2024-09-16T12:29:13.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:29:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13' 2024-09-16T12:29:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:29:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:29:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:29:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:29:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:29:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:29:13.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:29:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:29:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:29:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:29:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:29:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:29:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:29:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:29:13.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:29:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:29:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:29:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T12:29:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b9c65b6f-b177-4f69-b32f-158e1d24fdcc 2024-09-16T12:29:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13 2024-09-16T12:29:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:29:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:13.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13' 2024-09-16T12:29:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:29:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T12:29:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 56e0c85b-bcd6-45e9-baf8-ab981b2ca159 2024-09-16T12:29:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13 2024-09-16T12:29:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:13.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:29:13.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:13.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:13.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13' 2024-09-16T12:29:13.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T12:29:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 3a299e6e-0be4-4a52-9286-05028efdde9c 2024-09-16T12:29:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13 2024-09-16T12:29:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:29:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13' 2024-09-16T12:29:13.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:13.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T12:29:13.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: cc1f33be-5579-4d01-8ea8-2efb5bab2552 2024-09-16T12:29:13.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:13.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:13.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:13.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13 2024-09-16T12:29:13.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:13.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:29:13.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:13.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:13.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13' 2024-09-16T12:29:13.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T12:29:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a83eb5d8-a522-4c70-9ed3-260b39b46e23 2024-09-16T12:29:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13 2024-09-16T12:29:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:29:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13' 2024-09-16T12:29:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:29:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:29:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:29:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:29:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:29:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:29:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:29:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:29:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:29:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:29:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:29:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:29:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:29:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:29:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:29:13.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:29:13.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:29:13.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:29:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:29:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:29:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:29:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:29:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:29:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:13.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:29:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:29:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:29:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:29:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:29:13.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:29:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:29:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:29:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:29:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:29:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:29:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:29:14.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:29:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:29:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:29:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:29:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:29:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:29:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:29:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:29:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:29:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:29:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-16 12:29:13' 2024-09-16T12:29:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:29:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:29:13' 2024-09-16T12:29:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-16 12:29:13' 2024-09-16T12:29:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-16 12:29:13' 2024-09-16T12:29:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-16 12:29:13' 2024-09-16T12:29:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:29:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:29:14.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:29:13' 2024-09-16T12:29:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:29:13' 2024-09-16T12:29:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-16 12:29:13' 2024-09-16T12:29:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:29:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:29:13' 2024-09-16T12:29:14.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:29:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:29:13' 2024-09-16T12:29:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-16 12:29:13' 2024-09-16T12:29:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:29:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:29:13' 2024-09-16T12:29:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:14.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:29:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-16 12:29:13' 2024-09-16T12:29:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:29:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:29:13' 2024-09-16T12:29:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:29:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:29:14.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:29:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:29:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:29:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:14.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:29:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-16T12:29:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:29:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T12:29:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-16T12:29:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:29:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:14.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:14.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:14.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T12:29:14.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T12:29:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: d0aa1661-aaad-47bc-a934-2a5891ec1d70 2024-09-16T12:29:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13 2024-09-16T12:29:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:29:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13' 2024-09-16T12:29:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:14.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T12:29:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: eb4f75b8-2cf0-409a-b490-25d6035748ff 2024-09-16T12:29:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13 2024-09-16T12:29:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:29:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13' 2024-09-16T12:29:14.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:29:14.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13' 2024-09-16T12:29:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:29:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:29:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T12:29:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:29:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:29:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:29:14.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:29:14.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:14.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:29:14.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:29:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:14.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:29:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13' 2024-09-16T12:29:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:29:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:29:14.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T12:29:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:29:14.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:29:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:29:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13' 2024-09-16T12:29:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:29:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:29:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T12:29:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:29:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2<++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:29:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mirror_uuids>c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13' 2024-09-16T12:29:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:29:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:29:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T12:29:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:29:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13' 2024-09-16T12:29:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:29:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-16 12:29:13' 2024-09-16T12:29:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T12:29:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:29:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:29:13' 2024-09-16T12:29:14.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:29:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-16 12:29:13' 2024-09-16T12:29:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:29:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:29:13' 2024-09-16T12:29:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:14.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:29:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13' 2024-09-16T12:29:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:29:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T12:29:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13' 2024-09-16T12:29:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:29:14.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T12:29:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13' 2024-09-16T12:29:14.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:29:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T12:29:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13' 2024-09-16T12:29:14.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:29:14.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T12:29:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:14.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T12:29:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T12:29:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T12:29:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T12:29:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T12:29:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T12:29:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T12:29:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T12:29:14.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T12:29:14.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:29:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:29:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:29:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:29:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:29:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:29:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:29:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:29:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:29:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:29:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:29:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:29:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:29:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:29:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:29:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-16T12:29:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:29:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:29:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:14.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:14.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-16T12:29:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:29:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:29:14.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-16T12:29:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:29:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:29:14.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:29:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-16T12:29:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:29:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:29:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-16T12:29:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:29:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:29:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-16T12:29:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:29:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:29:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:29:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-16T12:29:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:29:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:29:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:29:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-16T12:29:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:29:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:29:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-16T12:29:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:29:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:29:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-16T12:29:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:29:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:14.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T12:29:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T12:29:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 72aa8251-8228-4686-a7e8-cfad99ad25c1 2024-09-16T12:29:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13 2024-09-16T12:29:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:29:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13' 2024-09-16T12:29:14.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:29:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:29:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:29:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:29:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:14.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:29:14.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:14.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:29:14.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:29:14.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:29:14.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:29:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-16 12:29:13' 2024-09-16T12:29:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:29:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:14.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:29:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:29:13' 2024-09-16T12:29:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:29:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:29:13cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:29:13' 2024-09-16T12:29:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:29:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T12:29:15.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T12:29:15.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:29:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:15.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:29:15.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-16T12:29:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:29:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:29:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:29:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-16T12:29:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:29:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-16T12:29:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:29:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-16T12:29:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:29:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-16T12:29:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-16T12:29:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:29:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:15.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-16T12:29:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-16T12:29:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-16T12:29:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:29:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-16T12:29:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-16T12:29:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:29:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:29:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-16T12:29:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-16T12:29:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:29:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-16T12:29:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:15.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-16T12:29:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-16T12:29:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:29:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-16T12:29:15.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-16T12:29:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:15.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-16T12:29:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T12:29:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: cc1f33be-5579-4d01-8ea8-2efb5bab2552 2024-09-16T12:29:15.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13 2024-09-16T12:29:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:29:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13' 2024-09-16T12:29:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:15.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:29:15.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:29:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T12:29:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: eb4f75b8-2cf0-409a-b490-25d6035748ff 2024-09-16T12:29:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13 2024-09-16T12:29:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:29:15.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13' 2024-09-16T12:29:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T12:29:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 9eb4b9fc-5335-4349-afe2-ff0112468825 2024-09-16T12:29:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13 2024-09-16T12:29:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:29:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13' 2024-09-16T12:29:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:15.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:29:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:29:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:29:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:29:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:29:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:29:15.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:29:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:29:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:29:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:29:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:29:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:29:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:29:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T12:29:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: d0aa1661-aaad-47bc-a934-2a5891ec1d70 2024-09-16T12:29:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13 2024-09-16T12:29:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:29:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13' 2024-09-16T12:29:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T12:29:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a83eb5d8-a522-4c70-9ed3-260b39b46e23 2024-09-16T12:29:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13 2024-09-16T12:29:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:29:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13' 2024-09-16T12:29:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T12:29:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 56e0c85b-bcd6-45e9-baf8-ab981b2ca159 2024-09-16T12:29:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13 2024-09-16T12:29:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:29:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13' 2024-09-16T12:29:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:15.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:29:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T12:29:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b9c65b6f-b177-4f69-b32f-158e1d24fdcc 2024-09-16T12:29:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13 2024-09-16T12:29:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:29:15.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13' 2024-09-16T12:29:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:29:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:29:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:29:15.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:29:15.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:29:15.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:29:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:29:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:29:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:29:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:29:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:29:15.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:29:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:29:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:29:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:29:15.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:29:15.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:29:15.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:29:15.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:29:15.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:29:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:29:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:29:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:29:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:29:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:29:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:29:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T12:29:15.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 3a299e6e-0be4-4a52-9286-05028efdde9c 2024-09-16T12:29:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13 2024-09-16T12:29:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:29:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13' 2024-09-16T12:29:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:15.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:29:15.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-16 12:29:13' 2024-09-16T12:29:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:29:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:29:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:29:13' 2024-09-16T12:29:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:29:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:29:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:29:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:29:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:29:15.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:29:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-16 12:29:13' 2024-09-16T12:29:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:29:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:29:13' 2024-09-16T12:29:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:15.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:29:15.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:15.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:29:15.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:29:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-16 12:29:13' 2024-09-16T12:29:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:29:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:29:13' 2024-09-16T12:29:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:29:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:29:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:29:15.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:29:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:29:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:29:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:29:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:29:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:29:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:29:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:29:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:29:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:29:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-16 12:29:13' 2024-09-16T12:29:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:29:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:29:13' 2024-09-16T12:29:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:15.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:29:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:29:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:29:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:29:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-16 12:29:13' 2024-09-16T12:29:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:29:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:29:13' 2024-09-16T12:29:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:29:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:29:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:29:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:29:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:29:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:29:15.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:29:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-16 12:29:13' 2024-09-16T12:29:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:29:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:29:13' 2024-09-16T12:29:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:29:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:15.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-16 12:29:13' 2024-09-16T12:29:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:29:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:29:13' 2024-09-16T12:29:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:29:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-16 12:29:13' 2024-09-16T12:29:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:29:15.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:29:13' 2024-09-16T12:29:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:15.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:29:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13' 2024-09-16T12:29:15.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:29:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T12:29:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T12:29:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:29:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:29:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-16T12:29:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:29:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:29:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-16T12:29:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13' 2024-09-16T12:29:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:29:15.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T12:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T12:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 76ea3943-f961-4a54-a62d-d75534b59757 2024-09-16T12:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13 2024-09-16T12:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:15.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13' 2024-09-16T12:29:15.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13' 2024-09-16T12:29:15.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:29:15.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T12:29:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:29:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:29:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T12:29:15.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:29:15.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:29:15.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:15.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:29:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:15.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:29:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T12:29:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:29:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:29:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:29:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:29:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:29:15.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:29:15.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:29:15.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-16 12:29:13' 2024-09-16T12:29:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:29:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:29:13' 2024-09-16T12:29:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:15.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:29:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-16T12:29:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:29:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:29:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-16T12:29:15.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:15.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-16T12:29:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:29:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:29:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-16T12:29:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13' 2024-09-16T12:29:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:29:15.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T12:29:15.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13' 2024-09-16T12:29:15.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:29:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T12:29:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13' 2024-09-16T12:29:15.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:29:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:15.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T12:29:15.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T12:29:15.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:29:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13' 2024-09-16T12:29:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:29:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T12:29:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:15.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:15.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:29:15.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T12:29:15.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-16T12:29:15.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:29:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:29:15.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-16T12:29:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:15.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:29:15.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13' 2024-09-16T12:29:15.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:29:15.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T12:29:15.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T12:29:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:29:15.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T12:29:15.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-16T12:29:15.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:29:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:29:15.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-16T12:29:15.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:29:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:15.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:29:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:29:15.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:29:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T12:29:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-16T12:29:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:29:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:29:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-16T12:29:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-16T12:29:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:29:15.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:29:15.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-16T12:29:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:29:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:29:15.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:15.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:15.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:15.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-16T12:29:15.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:29:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:29:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-16T12:29:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13' 2024-09-16T12:29:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:29:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T12:29:15.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:15.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T12:29:15.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:29:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:15.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:29:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-16T12:29:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:29:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:29:15.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-16T12:29:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-16T12:29:16.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:29:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:29:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:29:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:16.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-16T12:29:16.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:29:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:29:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:29:16.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:29:16.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:29:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:29:16.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:29:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T12:29:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 72aa8251-8228-4686-a7e8-cfad99ad25c1 2024-09-16T12:29:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:29:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13 2024-09-16T12:29:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:29:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:29:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:29:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:29:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:29:13' 2024-09-16T12:29:16.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:29:16.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:29:16.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:29:16.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:29:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary7 2024-09-16T12:29:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:29:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:29:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:29:16.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:29:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:29:16.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:16.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:29:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:29:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:29:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:29:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:29:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary6 2024-09-16T12:29:16.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary4 2024-09-16T12:29:16.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:29:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary1 2024-09-16T12:29:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:29:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:29:16.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary2 2024-09-16T12:29:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:29:16.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:29:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:29:16.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary8 2024-09-16T12:29:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-16 12:29:13' 2024-09-16T12:29:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:29:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:29:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:29:13' 2024-09-16T12:29:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:29:16.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary3 2024-09-16T12:29:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:29:16.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary5 2024-09-16T12:29:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary10 2024-09-16T12:29:16.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:29:16.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:29:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:29:13cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:29:13' 2024-09-16T12:29:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:29:16.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T12:29:16.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:29:13' 2024-09-16T12:29:16.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T12:29:16.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:29:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:29:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:29:16.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:29:16.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:16.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:29:16.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-16T12:29:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:29:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:29:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:29:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:29:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-16T12:29:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd0 2024-09-16T12:29:16.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:29:16.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd0 2024-09-16T12:29:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd1 2024-09-16T12:29:16.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:29:16.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd1 2024-09-16T12:29:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd2 2024-09-16T12:29:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:29:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd2 2024-09-16T12:29:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd3 2024-09-16T12:29:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:29:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd3 2024-09-16T12:29:16.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd4 2024-09-16T12:29:16.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:29:16.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd4 2024-09-16T12:29:16.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd5 2024-09-16T12:29:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:29:16.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd5 2024-09-16T12:29:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd6 2024-09-16T12:29:16.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd6 2024-09-16T12:29:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:29:16.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd7 2024-09-16T12:29:16.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:29:16.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd7 2024-09-16T12:29:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd8 2024-09-16T12:29:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd8 2024-09-16T12:29:16.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:29:17.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:29:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:29:17.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary9 2024-09-16T12:29:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd9 2024-09-16T12:29:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd9 2024-09-16T12:29:17.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:30:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=b256cce4b466352497b9199e4b4b07ab 2024-09-16T12:30:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd7 2024-09-16T12:30:52.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=3d7f357ec0731645da61d9d75ecc5bbf 2024-09-16T12:30:52.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd2 2024-09-16T12:30:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=60bb4de8cd16f3161de4094727c7c900 2024-09-16T12:30:52.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd8 2024-09-16T12:30:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=e57968b3040835636ac9d172b59254bc 2024-09-16T12:30:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-09-16T12:30:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=52bfc6783792e40af528c03b5a47f6eb 2024-09-16T12:30:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd3 2024-09-16T12:30:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=f3fe2949ca96e9fe275baa290be06617 2024-09-16T12:30:52.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd6 2024-09-16T12:30:52.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=80adcf0245b6cf843d097e456b523d5d 2024-09-16T12:30:52.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd1 2024-09-16T12:30:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=88b027a455ec7975b66ccab54701a7a2 2024-09-16T12:30:53.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd4 2024-09-16T12:30:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 3d7f357ec0731645da61d9d75ecc5bbf != \3\d\7\f\3\5\7\e\c\0\7\3\1\6\4\5\d\a\6\1\d\9\d\7\5\e\c\c\5\b\b\f ]] 2024-09-16T12:30:53.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=e427e1c4c7314e6400c4561097a65c40 2024-09-16T12:30:53.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd9 2024-09-16T12:30:53.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ b256cce4b466352497b9199e4b4b07ab != \b\2\5\6\c\c\e\4\b\4\6\6\3\5\2\4\9\7\b\9\1\9\9\e\4\b\4\b\0\7\a\b ]] 2024-09-16T12:30:53.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 60bb4de8cd16f3161de4094727c7c900 != \6\0\b\b\4\d\e\8\c\d\1\6\f\3\1\6\1\d\e\4\0\9\4\7\2\7\c\7\c\9\0\0 ]] 2024-09-16T12:30:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ e57968b3040835636ac9d172b59254bc != \e\5\7\9\6\8\b\3\0\4\0\8\3\5\6\3\6\a\c\9\d\1\7\2\b\5\9\2\5\4\b\c ]] 2024-09-16T12:30:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ f3fe2949ca96e9fe275baa290be06617 != \f\3\f\e\2\9\4\9\c\a\9\6\e\9\f\e\2\7\5\b\a\a\2\9\0\b\e\0\6\6\1\7 ]] 2024-09-16T12:30:53.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 52bfc6783792e40af528c03b5a47f6eb != \5\2\b\f\c\6\7\8\3\7\9\2\e\4\0\a\f\5\2\8\c\0\3\b\5\a\4\7\f\6\e\b ]] 2024-09-16T12:30:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 80adcf0245b6cf843d097e456b523d5d != \8\0\a\d\c\f\0\2\4\5\b\6\c\f\8\4\3\d\0\9\7\e\4\5\6\b\5\2\3\d\5\d ]] 2024-09-16T12:30:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:30:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 34847 2024-09-16T12:30:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:30:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 34848 2024-09-16T12:30:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:30:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 34851 2024-09-16T12:30:53.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=b5b5283b5350f653cbd2285551ca497f 2024-09-16T12:30:53.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd5 2024-09-16T12:30:54.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 88b027a455ec7975b66ccab54701a7a2 != \8\8\b\0\2\7\a\4\5\5\e\c\7\9\7\5\b\6\6\c\c\a\b\5\4\7\0\1\a\7\a\2 ]] 2024-09-16T12:30:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ e427e1c4c7314e6400c4561097a65c40 != \e\4\2\7\e\1\c\4\c\7\3\1\4\e\6\4\0\0\c\4\5\6\1\0\9\7\a\6\5\c\4\0 ]] 2024-09-16T12:30:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ b5b5283b5350f653cbd2285551ca497f != \b\5\b\5\2\8\3\b\5\3\5\0\f\6\5\3\c\b\d\2\2\8\5\5\5\1\c\a\4\9\7\f ]] 2024-09-16T12:30:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:30:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 34852 2024-09-16T12:30:54.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:30:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 34854 2024-09-16T12:30:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:30:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 34856 2024-09-16T12:30:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:30:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 34857 2024-09-16T12:30:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:30:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 34860 2024-09-16T12:30:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:30:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 34862 2024-09-16T12:30:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T12:30:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:30:54.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary1 2024-09-16T12:30:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-16T12:30:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:30:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:30:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:30:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:30:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:30:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:30:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:30:54.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:30:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-16T12:30:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:30:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:30:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:30:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:30:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:30:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:30:55.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:30:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:30:55.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T12:30:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T12:30:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: eb4f75b8-2cf0-409a-b490-25d6035748ff 2024-09-16T12:30:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:30:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":116985856.0,"last_snapshot_bytes":116985856,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"} 2024-09-16T12:30:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:30:55.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:30:53 2024-09-16T12:30:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:30:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:30:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:30:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:30:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:30:53' 2024-09-16T12:30:55.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:30:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:30:55.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:30:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":116985856.0,"last_snapshot_bytes":116985856,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"} 2024-09-16T12:30:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:30:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:30:55.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:30:55.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:30:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:30:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:30:55.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:30:55.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:30:55.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:30:53' 2024-09-16T12:30:55.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-16 12:30:53' 2024-09-16T12:30:55.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:30:55.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:30:55.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:30:55.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:30:55.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:30:53' 2024-09-16T12:30:55.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:30:55.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:30:55.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:30:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1eb4f75b8-2cf0-409a-b490-25d6035748ffup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":116985856.0,"last_snapshot_bytes":116985856,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:53image-primary1076ea3943-f961-4a54-a62d-d75534b59757up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:53image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":122658816.0,"last_snapshot_bytes":122658816,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:53image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70258688.0,"last_snapshot_bytes":70258688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:53image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70180864.0,"last_snapshot_bytes":70180864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:53image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70246400.0,"last_snapshot_bytes":70246400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:53image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":126840832.0,"last_snapshot_bytes":126840832,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:53image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":116338688.0,"last_snapshot_bytes":116338688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:53image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70152192.0,"last_snapshot_bytes":70152192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:53image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70316032.0,"last_snapshot_bytes":70316032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489757,"remote_snapshot_timestamp":1726489757,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:53' 2024-09-16T12:30:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:30:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T12:30:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:30:53' 2024-09-16T12:30:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T12:30:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:30:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:30:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:30:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:30:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:30:55.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:30:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-16T12:30:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:30:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:30:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:30:55.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-16T12:30:55.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-16T12:30:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:30:57.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-16T12:30:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:30:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:30:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:30:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:30:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:30:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary1 2024-09-16T12:30:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:30:57.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary1 2024-09-16T12:30:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:30:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary2 2024-09-16T12:30:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-16T12:30:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:30:57.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:30:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:30:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:30:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:30:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:30:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:30:57.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:30:58.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-16T12:30:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:30:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:30:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:30:58.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:30:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:30:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:30:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:30:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:30:58.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T12:30:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T12:30:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: d0aa1661-aaad-47bc-a934-2a5891ec1d70 2024-09-16T12:30:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:30:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":122658816.0,"last_snapshot_bytes":122658816,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"} 2024-09-16T12:30:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:30:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:30:53 2024-09-16T12:30:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:30:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:30:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:30:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:30:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:30:53' 2024-09-16T12:30:58.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:30:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:30:58.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:30:58.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":122658816.0,"last_snapshot_bytes":122658816,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"} 2024-09-16T12:30:58.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:30:58.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:30:58.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:30:58.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:30:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:30:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:30:58.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:30:58.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:30:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:30:53' 2024-09-16T12:30:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-16 12:30:53' 2024-09-16T12:30:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:30:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:30:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:30:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:30:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:30:53' 2024-09-16T12:30:58.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:30:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:30:58.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:30:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK942544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1076ea3943-f961-4a54-a62d-d75534b59757up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:53image-primary2d0aa1661-aaad-47bc-a934-2a5891ec1d70up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":122658816.0,"last_snapshot_bytes":122658816,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:58image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70258688.0,"last_snapshot_bytes":70258688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:58image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70180864.0,"last_snapshot_bytes":70180864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:58image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70246400.0,"last_snapshot_bytes":70246400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:58image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":126840832.0,"last_snapshot_bytes":126840832,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:58image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":116338688.0,"last_snapshot_bytes":116338688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:58image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70152192.0,"last_snapshot_bytes":70152192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:58image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70316032.0,"last_snapshot_bytes":70316032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489757,"remote_snapshot_timestamp":1726489757,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:58' 2024-09-16T12:30:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:30:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T12:30:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:30:53' 2024-09-16T12:30:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T12:30:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:30:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:30:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:30:58.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:30:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:30:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:30:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-16T12:30:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:30:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:30:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:30:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-16T12:30:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-16T12:31:00.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:31:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-16T12:31:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:00.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:31:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:31:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:31:00.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary2 2024-09-16T12:31:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:00.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary2 2024-09-16T12:31:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:31:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary3 2024-09-16T12:31:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-16T12:31:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:31:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:31:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:31:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:31:00.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:31:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-16T12:31:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:31:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:31:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:31:01.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:31:01.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:31:01.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T12:31:01.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T12:31:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: cc1f33be-5579-4d01-8ea8-2efb5bab2552 2024-09-16T12:31:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:31:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70258688.0,"last_snapshot_bytes":70258688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"} 2024-09-16T12:31:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:31:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:30:53 2024-09-16T12:31:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:31:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:31:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:31:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:31:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:30:58' 2024-09-16T12:31:01.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:31:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:31:01.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:31:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70258688.0,"last_snapshot_bytes":70258688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"} 2024-09-16T12:31:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:31:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:31:01.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:31:01.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:31:01.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:31:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:31:01.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:31:01.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:31:01.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:30:53' 2024-09-16T12:31:01.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-16 12:30:53' 2024-09-16T12:31:01.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:31:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:30:53' 2024-09-16T12:31:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:31:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:31:01.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:31:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK842544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1076ea3943-f961-4a54-a62d-d75534b59757up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:58image-primary3cc1f33be-5579-4d01-8ea8-2efb5bab2552up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70258688.0,"last_snapshot_bytes":70258688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:58image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70180864.0,"last_snapshot_bytes":70180864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:58image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70246400.0,"last_snapshot_bytes":70246400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:58image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":126840832.0,"last_snapshot_bytes":126840832,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:58image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":116338688.0,"last_snapshot_bytes":116338688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:58image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70152192.0,"last_snapshot_bytes":70152192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:58image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70316032.0,"last_snapshot_bytes":70316032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489757,"remote_snapshot_timestamp":1726489757,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:30:58' 2024-09-16T12:31:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:31:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T12:31:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:30:53' 2024-09-16T12:31:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T12:31:01.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:31:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:31:01.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:31:01.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:31:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-16T12:31:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:31:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:31:01.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-16T12:31:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-16T12:31:03.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:31:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-16T12:31:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:03.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:31:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:31:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:31:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary3 2024-09-16T12:31:03.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:03.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary3 2024-09-16T12:31:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:31:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary4 2024-09-16T12:31:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-16T12:31:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:31:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:31:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:31:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:31:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:31:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-16T12:31:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:31:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:31:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:31:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:31:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:31:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T12:31:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T12:31:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a83eb5d8-a522-4c70-9ed3-260b39b46e23 2024-09-16T12:31:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:31:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70180864.0,"last_snapshot_bytes":70180864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"} 2024-09-16T12:31:04.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:31:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:30:53 2024-09-16T12:31:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:31:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:31:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:31:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:31:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:31:03' 2024-09-16T12:31:04.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:31:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:31:04.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:31:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70180864.0,"last_snapshot_bytes":70180864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"} 2024-09-16T12:31:04.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:31:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:31:04.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:31:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:31:04.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:31:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:31:04.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:31:04.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:31:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:30:53' 2024-09-16T12:31:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-16 12:30:53' 2024-09-16T12:31:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:31:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:30:53' 2024-09-16T12:31:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:31:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:31:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:31:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK742544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1076ea3943-f961-4a54-a62d-d75534b59757up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:03image-primary4a83eb5d8-a522-4c70-9ed3-260b39b46e23up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70180864.0,"last_snapshot_bytes":70180864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:03image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70246400.0,"last_snapshot_bytes":70246400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:03image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":126840832.0,"last_snapshot_bytes":126840832,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:03image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":116338688.0,"last_snapshot_bytes":116338688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:03image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70152192.0,"last_snapshot_bytes":70152192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:03image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70316032.0,"last_snapshot_bytes":70316032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489757,"remote_snapshot_timestamp":1726489757,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:03' 2024-09-16T12:31:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:31:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T12:31:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:30:53' 2024-09-16T12:31:04.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T12:31:04.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:31:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:31:04.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:31:04.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:31:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-16T12:31:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:31:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:31:04.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-16T12:31:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-16T12:31:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:31:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-16T12:31:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:31:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:31:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary4 2024-09-16T12:31:06.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:31:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary4 2024-09-16T12:31:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:31:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary5 2024-09-16T12:31:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-16T12:31:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:31:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:31:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:31:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:31:06.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:31:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-16T12:31:07.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:31:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:31:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:31:07.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:31:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:31:07.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T12:31:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T12:31:07.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 3a299e6e-0be4-4a52-9286-05028efdde9c 2024-09-16T12:31:07.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:31:07.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70246400.0,"last_snapshot_bytes":70246400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"} 2024-09-16T12:31:07.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:31:07.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:30:53 2024-09-16T12:31:07.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:31:07.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:31:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:31:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:31:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:31:03' 2024-09-16T12:31:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:31:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:31:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:31:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70246400.0,"last_snapshot_bytes":70246400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"} 2024-09-16T12:31:07.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:31:07.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:31:07.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:31:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:31:07.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:31:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:31:07.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:31:07.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:31:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:30:53' 2024-09-16T12:31:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-16 12:30:53' 2024-09-16T12:31:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:07.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:31:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:30:53' 2024-09-16T12:31:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:31:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:31:07.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:31:07.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK642544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1076ea3943-f961-4a54-a62d-d75534b59757up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:03image-primary53a299e6e-0be4-4a52-9286-05028efdde9cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70246400.0,"last_snapshot_bytes":70246400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:03image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":126840832.0,"last_snapshot_bytes":126840832,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:03image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":116338688.0,"last_snapshot_bytes":116338688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:03image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70152192.0,"last_snapshot_bytes":70152192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:03image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70316032.0,"last_snapshot_bytes":70316032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489757,"remote_snapshot_timestamp":1726489757,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:03' 2024-09-16T12:31:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:31:07.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T12:31:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:30:53' 2024-09-16T12:31:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T12:31:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:31:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:31:07.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:31:07.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:31:07.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:07.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:07.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-16T12:31:07.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:31:07.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:07.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:31:07.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-16T12:31:07.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-16T12:31:09.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:31:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-16T12:31:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:31:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:31:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:31:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary5 2024-09-16T12:31:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary5 2024-09-16T12:31:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:31:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary6 2024-09-16T12:31:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-16T12:31:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:31:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:31:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:31:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:31:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:31:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-16T12:31:10.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:31:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:31:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:31:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:31:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:31:10.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T12:31:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T12:31:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 56e0c85b-bcd6-45e9-baf8-ab981b2ca159 2024-09-16T12:31:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:31:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":126840832.0,"last_snapshot_bytes":126840832,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"} 2024-09-16T12:31:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:31:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:30:53 2024-09-16T12:31:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:31:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:31:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:31:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:31:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:31:08' 2024-09-16T12:31:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:31:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:31:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:31:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":126840832.0,"last_snapshot_bytes":126840832,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"} 2024-09-16T12:31:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:31:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:31:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:31:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:31:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:31:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:31:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:31:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:31:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:30:53' 2024-09-16T12:31:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-16 12:30:53' 2024-09-16T12:31:10.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:31:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:30:53' 2024-09-16T12:31:10.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:31:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:31:10.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:31:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK542544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1076ea3943-f961-4a54-a62d-d75534b59757up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:08image-primary656e0c85b-bcd6-45e9-baf8-ab981b2ca159up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":126840832.0,"last_snapshot_bytes":126840832,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:08image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":116338688.0,"last_snapshot_bytes":116338688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:08image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70152192.0,"last_snapshot_bytes":70152192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:08image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70316032.0,"last_snapshot_bytes":70316032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489757,"remote_snapshot_timestamp":1726489757,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:08' 2024-09-16T12:31:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:31:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T12:31:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:30:53' 2024-09-16T12:31:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T12:31:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:31:10.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:31:10.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:31:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:31:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-16T12:31:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:31:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:31:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-16T12:31:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-16T12:31:12.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:31:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-16T12:31:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:31:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:31:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:31:12.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary6 2024-09-16T12:31:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:12.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary6 2024-09-16T12:31:12.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:31:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary7 2024-09-16T12:31:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-16T12:31:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:31:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:31:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:31:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:31:12.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:31:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-16T12:31:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:31:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:31:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:31:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:31:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:31:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T12:31:13.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T12:31:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b9c65b6f-b177-4f69-b32f-158e1d24fdcc 2024-09-16T12:31:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:31:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":116338688.0,"last_snapshot_bytes":116338688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"} 2024-09-16T12:31:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:31:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:30:53 2024-09-16T12:31:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:31:13.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:31:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:31:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:31:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:31:08' 2024-09-16T12:31:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:31:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:31:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:31:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":116338688.0,"last_snapshot_bytes":116338688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"} 2024-09-16T12:31:13.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:31:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:31:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:31:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:31:13.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:31:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:31:13.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:31:13.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:31:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:30:53' 2024-09-16T12:31:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-16 12:30:53' 2024-09-16T12:31:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:31:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:30:53' 2024-09-16T12:31:13.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:31:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:31:13.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:31:13.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK442544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1076ea3943-f961-4a54-a62d-d75534b59757up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:13image-primary7b9c65b6f-b177-4f69-b32f-158e1d24fdccup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":116338688.0,"last_snapshot_bytes":116338688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70152192.0,"last_snapshot_bytes":70152192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70316032.0,"last_snapshot_bytes":70316032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489757,"remote_snapshot_timestamp":1726489757,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:13' 2024-09-16T12:31:13.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:31:13.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T12:31:13.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:30:53' 2024-09-16T12:31:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T12:31:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:31:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:31:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:31:13.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:31:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-16T12:31:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:31:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:31:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-16T12:31:13.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-16T12:31:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:31:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-16T12:31:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:31:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:31:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:31:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary7 2024-09-16T12:31:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary7 2024-09-16T12:31:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:31:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary8 2024-09-16T12:31:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-16T12:31:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:31:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:31:15.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:31:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:31:15.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:31:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-16T12:31:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:31:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:31:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:31:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:31:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:31:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T12:31:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T12:31:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 9eb4b9fc-5335-4349-afe2-ff0112468825 2024-09-16T12:31:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:31:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70152192.0,"last_snapshot_bytes":70152192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"} 2024-09-16T12:31:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:31:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:30:53 2024-09-16T12:31:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:31:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:31:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:31:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:31:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:31:13' 2024-09-16T12:31:16.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:31:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:31:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:31:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70152192.0,"last_snapshot_bytes":70152192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"} 2024-09-16T12:31:16.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:31:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:31:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:31:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:31:16.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:31:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:31:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:31:16.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:31:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:30:53' 2024-09-16T12:31:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-16 12:30:53' 2024-09-16T12:31:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:31:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:30:53' 2024-09-16T12:31:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:31:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:31:16.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:31:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK342544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1076ea3943-f961-4a54-a62d-d75534b59757up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:13image-primary89eb4b9fc-5335-4349-afe2-ff0112468825up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70152192.0,"last_snapshot_bytes":70152192,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:13image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70316032.0,"last_snapshot_bytes":70316032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489757,"remote_snapshot_timestamp":1726489757,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:13' 2024-09-16T12:31:16.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:31:16.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T12:31:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:30:53' 2024-09-16T12:31:16.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T12:31:16.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:31:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:31:16.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:31:16.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:31:16.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:16.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:16.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-16T12:31:16.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:31:16.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:16.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:31:16.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-16T12:31:16.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-16T12:31:18.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:31:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-16T12:31:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:31:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:31:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:31:18.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary8 2024-09-16T12:31:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary8 2024-09-16T12:31:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:31:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary9 2024-09-16T12:31:18.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-16T12:31:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:31:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:31:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:31:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:31:18.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:31:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-16T12:31:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:31:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:19.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:31:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:31:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:31:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:31:19.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T12:31:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T12:31:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 72aa8251-8228-4686-a7e8-cfad99ad25c1 2024-09-16T12:31:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:31:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70316032.0,"last_snapshot_bytes":70316032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489757,"remote_snapshot_timestamp":1726489757,"replay_state":"idle"} 2024-09-16T12:31:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:31:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:30:53 2024-09-16T12:31:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:31:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:31:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:31:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:31:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:31:18' 2024-09-16T12:31:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:31:19.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:31:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:31:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70316032.0,"last_snapshot_bytes":70316032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489757,"remote_snapshot_timestamp":1726489757,"replay_state":"idle"} 2024-09-16T12:31:19.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:31:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:31:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:31:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:31:19.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:31:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:31:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:31:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:31:19.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:30:53' 2024-09-16T12:31:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-16 12:30:53' 2024-09-16T12:31:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:31:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:30:53' 2024-09-16T12:31:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:31:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:31:19.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:31:19.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK242544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1076ea3943-f961-4a54-a62d-d75534b59757up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:18image-primary972aa8251-8228-4686-a7e8-cfad99ad25c1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70316032.0,"last_snapshot_bytes":70316032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489757,"remote_snapshot_timestamp":1726489757,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:18' 2024-09-16T12:31:19.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:31:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T12:31:19.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:30:53' 2024-09-16T12:31:19.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T12:31:19.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:31:19.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:31:19.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:31:19.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:31:19.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:19.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:19.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-16T12:31:19.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:31:19.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:19.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:31:19.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-16T12:31:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-16T12:31:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:31:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-16T12:31:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:31:21.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:31:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:31:21.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary9 2024-09-16T12:31:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary9 2024-09-16T12:31:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:31:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary10 2024-09-16T12:31:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-16T12:31:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:21.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:31:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:31:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:31:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:31:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:31:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-16T12:31:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:22.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:31:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:31:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:31:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:31:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:31:22.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T12:31:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T12:31:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 76ea3943-f961-4a54-a62d-d75534b59757 2024-09-16T12:31:22.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:31:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"} 2024-09-16T12:31:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:31:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:30:53 2024-09-16T12:31:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:31:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:31:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:31:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:31:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:31:18' 2024-09-16T12:31:22.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:31:22.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:31:22.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:31:22.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"} 2024-09-16T12:31:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:31:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:31:22.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:31:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:31:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:31:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:31:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:31:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:31:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:30:53' 2024-09-16T12:31:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-16 12:30:53' 2024-09-16T12:31:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:31:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:31:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:30:53' 2024-09-16T12:31:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:31:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:31:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:31:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK142544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1076ea3943-f961-4a54-a62d-d75534b59757up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726489756,"remote_snapshot_timestamp":1726489756,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:31:18' 2024-09-16T12:31:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:31:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T12:31:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:30:53' 2024-09-16T12:31:22.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T12:31:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:31:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:31:22.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:31:22.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:31:22.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-16T12:31:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:31:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:31:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-16T12:31:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-16T12:31:24.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:31:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-16T12:31:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:31:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:31:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:31:24.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary10 2024-09-16T12:31:24.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:31:24.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary10 2024-09-16T12:31:24.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..10} 2024-09-16T12:31:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS=() 2024-09-16T12:31:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS=() 2024-09-16T12:31:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS=() 2024-09-16T12:31:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS=() 2024-09-16T12:31:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ RET=0 2024-09-16T12:31:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:31:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary1 2024-09-16T12:31:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary1 2024-09-16T12:31:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-16T12:31:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:31:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:31:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:31:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-16T12:31:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:31:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:24.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:31:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:31:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:31:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-16T12:31:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:31:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-16T12:31:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:31:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:31:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-16T12:31:24.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:31:24.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-16T12:31:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary1': 2024-09-16T12:31:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:31:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:31:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:31:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 15571151c7e7 2024-09-16T12:31:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.15571151c7e7 2024-09-16T12:31:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:31:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:31:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:31:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:31:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:31:24 2024 2024-09-16T12:31:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:31:24 2024 2024-09-16T12:31:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:31:24 2024 2024-09-16T12:31:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:31:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:31:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: dba8131a-5e5e-4a91-bc7c-b81e67b290cd 2024-09-16T12:31:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:31:24.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:31:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary1 2024-09-16T12:31:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd0 2024-09-16T12:31:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:31:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd0 2024-09-16T12:31:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:31:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:31:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:31:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 5dc5c375-b431-48f3-9405-7f442a67e7f9 2024-09-16T12:31:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:31:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:31:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:31:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:31:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:31:25.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:31:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:31:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:31:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary1 2024-09-16T12:31:25.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd0 test-primary1 2024-09-16T12:31:25.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:31:25.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary1 2024-09-16T12:31:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:31:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:31:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-16T12:31:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:31:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary2 2024-09-16T12:31:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary2 2024-09-16T12:31:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-16T12:31:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:31:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:31:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:31:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:31:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:31:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary1 2024-09-16T12:31:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-16T12:31:25.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:31:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:31:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:31:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:31:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:31:25.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:31:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-16T12:31:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary1 2024-09-16T12:31:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-16T12:31:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:31:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-16T12:31:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:25.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:31:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:31:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-16T12:31:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:31:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-16T12:31:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 659 2024-09-16T12:31:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:31:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:25.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-16T12:31:25.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary2': 2024-09-16T12:31:25.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:31:25.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:31:25.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:31:25.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 156455bdc644 2024-09-16T12:31:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.156455bdc644 2024-09-16T12:31:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:31:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:31:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:31:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:31:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:31:25 2024 2024-09-16T12:31:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:31:25 2024 2024-09-16T12:31:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:31:25 2024 2024-09-16T12:31:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:31:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:31:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 513096d2-ff05-4526-8ac4-746ec0e8d36c 2024-09-16T12:31:25.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:31:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:31:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary2 2024-09-16T12:31:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd1 2024-09-16T12:31:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:31:26.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd1 2024-09-16T12:31:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:31:26.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:31:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:31:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: f0ba9518-4751-4ee7-88af-e93685d11960 2024-09-16T12:31:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:31:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:31:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:31:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:31:26.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:31:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:31:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:31:26.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:31:26.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary2 2024-09-16T12:31:26.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd1 test-primary2 2024-09-16T12:31:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:31:26.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary2 2024-09-16T12:31:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:31:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-16T12:31:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:31:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary2 2024-09-16T12:31:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:31:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:31:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary3 2024-09-16T12:31:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary2 2024-09-16T12:31:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:31:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary3 2024-09-16T12:31:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-16T12:31:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-16T12:31:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:31:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:31:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:31:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:31:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:31:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-16T12:31:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:31:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:31:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:31:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:31:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-16T12:31:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:31:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-16T12:31:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:26.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:31:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:31:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-16T12:31:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:31:26.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-16T12:31:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 661 2024-09-16T12:31:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:31:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-16T12:31:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary3': 2024-09-16T12:31:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:31:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:31:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:31:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1576f18653af 2024-09-16T12:31:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1576f18653af 2024-09-16T12:31:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:31:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:31:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:31:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:31:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:31:26 2024 2024-09-16T12:31:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:31:26 2024 2024-09-16T12:31:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:31:26 2024 2024-09-16T12:31:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:31:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:31:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 2d9ab414-6b90-42a5-a42c-00a9400d330b 2024-09-16T12:31:26.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:31:26.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:31:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary3 2024-09-16T12:31:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd2 2024-09-16T12:31:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:31:26.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd2 2024-09-16T12:31:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:31:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 528384/2621440 done 2024-09-16T12:31:27.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:31:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 59e09650-6110-4ed9-a1b2-924fad33c6a0 2024-09-16T12:31:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:31:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:31:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:31:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:31:27.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:31:27.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:31:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:31:27.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:31:27.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary3 2024-09-16T12:31:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd2 test-primary3 2024-09-16T12:31:27.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:31:27.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary3 2024-09-16T12:31:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:31:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-16T12:31:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:31:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:31:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:31:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:31:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary4 2024-09-16T12:31:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary4 2024-09-16T12:31:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:31:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-16T12:31:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:31:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:27.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:31:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:31:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:31:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-16T12:31:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:31:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:27.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:31:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:31:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:31:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-16T12:31:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary3 2024-09-16T12:31:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary3 2024-09-16T12:31:27.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-16T12:31:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:31:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-16T12:31:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:31:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:31:27.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-16T12:31:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:31:27.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-16T12:31:27.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:31:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 664 2024-09-16T12:31:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-16T12:31:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary4': 2024-09-16T12:31:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:31:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:31:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:31:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 15852751613a 2024-09-16T12:31:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.15852751613a 2024-09-16T12:31:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:31:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:31:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:31:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:31:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:31:27 2024 2024-09-16T12:31:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:31:27 2024 2024-09-16T12:31:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:31:27 2024 2024-09-16T12:31:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:31:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:31:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 7c22ff11-90eb-4a64-9753-ee0488c7c6f0 2024-09-16T12:31:27.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:31:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:31:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary4 2024-09-16T12:31:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd3 2024-09-16T12:31:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:31:27.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd3 2024-09-16T12:31:27.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:31:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 528384/2621440 done 2024-09-16T12:31:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:31:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 34965ce6-b5cb-480a-a313-f79be4f6c4be 2024-09-16T12:31:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:31:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:31:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:31:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:31:28.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:31:28.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:31:28.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:31:28.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:31:28.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary4 2024-09-16T12:31:28.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd3 test-primary4 2024-09-16T12:31:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:31:28.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary4 2024-09-16T12:31:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:31:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:31:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:31:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary5 2024-09-16T12:31:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary5 2024-09-16T12:31:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-16T12:31:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:31:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:31:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:31:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-16T12:31:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:28.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:31:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-16T12:31:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:31:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:31:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:31:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-16T12:31:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:31:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:31:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:28.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:31:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:31:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary4 2024-09-16T12:31:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary4 2024-09-16T12:31:28.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-16T12:31:28.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:31:28.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-16T12:31:28.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:28.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:28.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:31:28.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:31:28.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-16T12:31:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:31:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-16T12:31:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 667 2024-09-16T12:31:28.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:31:28.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:31:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:31:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:31:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-16T12:31:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary5': 2024-09-16T12:31:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:31:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:31:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:31:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 15935565202a 2024-09-16T12:31:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.15935565202a 2024-09-16T12:31:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:31:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:31:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:31:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:31:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:31:28 2024 2024-09-16T12:31:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:31:28 2024 2024-09-16T12:31:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:31:28 2024 2024-09-16T12:31:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:31:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:31:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: b2fd01b8-9e0f-4858-992e-38984ef0b6a3 2024-09-16T12:31:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:31:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:31:29.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary5 2024-09-16T12:31:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd4 2024-09-16T12:31:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:31:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd4 2024-09-16T12:31:29.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:31:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:31:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:31:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: d47e843f-e162-4365-b608-f8b29062ae86 2024-09-16T12:31:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:31:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:31:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:31:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:31:29.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:31:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:31:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:31:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:31:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary5 2024-09-16T12:31:29.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd4 test-primary5 2024-09-16T12:31:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:31:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary5 2024-09-16T12:31:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:31:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-16T12:31:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:31:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:31:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:31:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:31:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:31:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary5 2024-09-16T12:31:29.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:31:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary6 2024-09-16T12:31:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary6 2024-09-16T12:31:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary5 2024-09-16T12:31:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-16T12:31:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-16T12:31:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:31:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:31:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:31:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-16T12:31:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:31:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:31:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:31:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:31:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-16T12:31:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:31:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:31:29.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:31:29.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:31:29.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-16T12:31:29.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:29.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:29.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:31:29.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:31:29.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-16T12:31:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:31:29.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-16T12:31:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 670 2024-09-16T12:31:30.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:31:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:31:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:31:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 671 2024-09-16T12:31:30.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:31:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:30.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-16T12:31:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary6': 2024-09-16T12:31:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:31:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:31:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:31:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 15a6b404cc60 2024-09-16T12:31:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.15a6b404cc60 2024-09-16T12:31:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:31:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:31:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:31:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:31:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:31:29 2024 2024-09-16T12:31:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:31:29 2024 2024-09-16T12:31:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:31:29 2024 2024-09-16T12:31:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:31:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:31:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 1166f0a7-946b-4553-bdb8-54616925dac3 2024-09-16T12:31:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:31:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:31:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary6 2024-09-16T12:31:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd5 2024-09-16T12:31:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:31:31.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd5 2024-09-16T12:31:31.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:31:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:31:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:31:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 926ed742-0a67-498e-9e8d-333dba3a4124 2024-09-16T12:31:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:31:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:31:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:31:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:31:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:31:31.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:31:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:31:31.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:31:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary6 2024-09-16T12:31:31.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd5 test-primary6 2024-09-16T12:31:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:31:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary6 2024-09-16T12:31:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:31:31.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-16T12:31:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:31:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:31:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:31:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:31:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:31:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary6 2024-09-16T12:31:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:31:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary7 2024-09-16T12:31:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary7 2024-09-16T12:31:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-16T12:31:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary6 2024-09-16T12:31:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-16T12:31:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:31:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:31:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:31:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-16T12:31:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:31:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:31:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:31:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:31:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-16T12:31:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:31:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-16T12:31:31.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:31:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:31:31.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-16T12:31:31.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:31:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-16T12:31:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 673 2024-09-16T12:31:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:31:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:31:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:31:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:31.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 678 2024-09-16T12:31:31.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:31.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:31:31.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-16T12:31:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary7': 2024-09-16T12:31:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:31:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:31:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:31:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 15bcc8c7cac2 2024-09-16T12:31:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.15bcc8c7cac2 2024-09-16T12:31:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:31:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:31:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:31:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:31:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:31:31 2024 2024-09-16T12:31:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:31:31 2024 2024-09-16T12:31:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:31:31 2024 2024-09-16T12:31:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:31:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:31:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 84b20afe-0f64-4172-b43d-9f6a37443624 2024-09-16T12:31:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:31:31.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:31:31.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary7 2024-09-16T12:31:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd6 2024-09-16T12:31:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:31:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd6 2024-09-16T12:31:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:31:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:31:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:31:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 8e4fa324-a863-497f-b200-927e33e5fbab 2024-09-16T12:31:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:31:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:31:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:31:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:31:32.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:31:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:31:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:31:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:31:32.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary7 2024-09-16T12:31:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd6 test-primary7 2024-09-16T12:31:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:31:32.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary7 2024-09-16T12:31:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:31:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-16T12:31:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:31:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:31:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:31:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary8 2024-09-16T12:31:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary8 2024-09-16T12:31:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-16T12:31:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary7 2024-09-16T12:31:32.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:31:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:31:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:31:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:31:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:31:32.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:31:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-16T12:31:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary7 2024-09-16T12:31:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-16T12:31:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:31:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:31:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:31:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:31:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-16T12:31:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:31:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-16T12:31:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:31:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:31:32.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-16T12:31:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:31:32.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-16T12:31:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 676 2024-09-16T12:31:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:32.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 683 2024-09-16T12:31:32.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:31:32.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:32.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-16T12:31:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary8': 2024-09-16T12:31:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:31:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:31:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:31:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 15ccf115f285 2024-09-16T12:31:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.15ccf115f285 2024-09-16T12:31:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:31:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:31:32.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:31:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:31:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:31:32 2024 2024-09-16T12:31:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:31:32 2024 2024-09-16T12:31:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:31:32 2024 2024-09-16T12:31:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:31:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:31:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 03a4f57d-6ac5-4d45-8468-694f291bfc5a 2024-09-16T12:31:32.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:31:32.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:31:32.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary8 2024-09-16T12:31:32.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd7 2024-09-16T12:31:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:31:32.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd7 2024-09-16T12:31:32.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:31:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/26214402101248/2621440 done 2024-09-16T12:31:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:31:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 5428ecdd-d823-4472-b088-996f866ed5e5 2024-09-16T12:31:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:31:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:31:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:31:33.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:31:33.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:31:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:31:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:31:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:31:33.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary8 2024-09-16T12:31:33.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd7 test-primary8 2024-09-16T12:31:33.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:31:33.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary8 2024-09-16T12:31:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:31:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-16T12:31:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:31:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:31:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:31:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary9 2024-09-16T12:31:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary9 2024-09-16T12:31:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary8 2024-09-16T12:31:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:31:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-16T12:31:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:33.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:31:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:31:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:31:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary8 2024-09-16T12:31:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:31:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:31:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-16T12:31:33.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-16T12:31:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:31:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:31:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:31:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:31:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:33.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-16T12:31:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:31:33.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-16T12:31:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:31:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:31:33.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-16T12:31:33.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:31:33.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-16T12:31:33.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 682 2024-09-16T12:31:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:33.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:31:33.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 687 2024-09-16T12:31:33.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:33.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-16T12:31:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:31:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:31:33.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:31:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:31:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:31:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:31:33.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary9': 2024-09-16T12:31:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:31:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:31:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:31:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 15df7ee6a7db 2024-09-16T12:31:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.15df7ee6a7db 2024-09-16T12:31:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:31:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:31:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:31:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:31:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:31:33 2024 2024-09-16T12:31:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:31:33 2024 2024-09-16T12:31:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:31:33 2024 2024-09-16T12:31:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:31:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:31:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 6588a7f2-9bf0-4c8d-9a72-4664809bda6b 2024-09-16T12:31:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:31:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:31:33.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary9 2024-09-16T12:31:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd8 2024-09-16T12:31:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:31:33.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd8 2024-09-16T12:31:33.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 77782428-d44f-4d0e-9117-d54fb2adecdb 2024-09-16T12:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:31:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:31:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:31:34.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:31:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:31:34.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:31:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary9 2024-09-16T12:31:34.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd8 test-primary9 2024-09-16T12:31:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:31:34.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary9 2024-09-16T12:31:34.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:31:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-16T12:31:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:31:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:31:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:31:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:31:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:31:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:31:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary9 2024-09-16T12:31:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary10 2024-09-16T12:31:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary10 2024-09-16T12:31:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-16T12:31:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary9 2024-09-16T12:31:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:31:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-16T12:31:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:31:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:31:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-16T12:31:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:31:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:31:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:31:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:31:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:31:34.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-16T12:31:34.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:31:34.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-16T12:31:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:31:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:31:34.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-16T12:31:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:31:34.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-16T12:31:34.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 691 2024-09-16T12:31:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:31:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:31:34.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:31:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:34.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:31:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:31:34.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:31:35.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 690 2024-09-16T12:31:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:31:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:31:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:31:35.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 689 2024-09-16T12:31:35.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:31:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:31:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:31:35.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:31:35.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-16T12:31:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary10': 2024-09-16T12:31:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:31:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:31:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:31:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 15f4536b3aa3 2024-09-16T12:31:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.15f4536b3aa3 2024-09-16T12:31:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:31:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:31:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:31:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:31:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:31:34 2024 2024-09-16T12:31:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:31:34 2024 2024-09-16T12:31:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:31:34 2024 2024-09-16T12:31:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:31:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:31:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: f67b4d08-9215-4c3a-be5a-c06ea7bbbb73 2024-09-16T12:31:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:31:35.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:31:35.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary10 2024-09-16T12:31:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd9 2024-09-16T12:31:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:31:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd9 2024-09-16T12:31:36.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:31:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:31:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:31:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 6f113645-a13f-4063-bf7b-ae3101c0fd26 2024-09-16T12:31:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:31:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:31:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:31:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:31:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:31:36.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:31:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:31:36.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:31:36.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary10 2024-09-16T12:31:36.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd9 test-primary10 2024-09-16T12:31:36.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:31:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary10 2024-09-16T12:31:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:31:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:31:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:31:36.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 39285 2024-09-16T12:31:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary10 2024-09-16T12:31:36.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary10 2024-09-16T12:31:36.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-16T12:31:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-16T12:31:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:31:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:31:36.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:31:36.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:31:36.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 696 2024-09-16T12:31:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 695 2024-09-16T12:31:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:31:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:31:36.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:31:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:31:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-16T12:31:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:31:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:31:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:31:37.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 698 2024-09-16T12:31:37.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:37.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 700 2024-09-16T12:31:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:31:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:31:37.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:31:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 701 2024-09-16T12:31:37.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:38.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 705 2024-09-16T12:31:38.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 704 2024-09-16T12:31:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:31:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:31:38.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:31:38.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:38.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:38.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:31:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:31:38.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:31:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 708 2024-09-16T12:31:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:31:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:31:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:31:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:31:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:31:39.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:31:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:31:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:31:40.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:31:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 710 2024-09-16T12:31:40.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 711 2024-09-16T12:31:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:31:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:31:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:31:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:31:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:31:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:31:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:31:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:31:41.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:31:41.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:41.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:31:41.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:41.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:41.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:31:41.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:31:41.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 715 2024-09-16T12:31:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:41.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 714 2024-09-16T12:31:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:42.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 719 2024-09-16T12:31:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 718 2024-09-16T12:31:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 720 2024-09-16T12:31:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:31:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:42.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:31:42.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:31:43.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 723 2024-09-16T12:31:43.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 724 2024-09-16T12:31:43.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:31:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:31:43.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:31:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:31:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:43.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:31:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:31:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 728 2024-09-16T12:31:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:31:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:31:44.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:31:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:31:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:31:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:31:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:31:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:31:45.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:31:45.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:31:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:31:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:31:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:31:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:31:45.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:31:45.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 731 2024-09-16T12:31:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 730 2024-09-16T12:31:45.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:31:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:31:46.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:31:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 735 2024-09-16T12:31:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 734 2024-09-16T12:31:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:31:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:31:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:31:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:47.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:47.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:31:47.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:31:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:31:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 738 2024-09-16T12:31:47.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 739 2024-09-16T12:31:47.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:47.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:47.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 736 2024-09-16T12:31:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:31:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:31:48.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:31:48.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:31:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:31:48.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:31:48.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 745 2024-09-16T12:31:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 743 2024-09-16T12:31:48.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:48.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 748 2024-09-16T12:31:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:31:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:31:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:31:49.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:31:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:31:49.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:31:50.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 751 2024-09-16T12:31:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 750 2024-09-16T12:31:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:31:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:31:50.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:31:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:31:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:31:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:31:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:31:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:31:50.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:31:51.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 754 2024-09-16T12:31:51.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:51.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:31:51.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:31:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:31:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:31:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:31:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:31:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 755 2024-09-16T12:31:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:52.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 759 2024-09-16T12:31:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 758 2024-09-16T12:31:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 760 2024-09-16T12:31:52.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:52.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:52.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:31:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:52.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:31:52.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:31:53.538 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:31:53.529+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e36aa000 2024-09-16T12:31:53.538 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:31:53.529+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e36ab1e0 2024-09-16T12:31:53.541 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:31:53.529+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e5d9cf70 2024-09-16T12:31:53.541 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:31:53.529+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e5d9d2b0 2024-09-16T12:31:53.541 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:31:53.529+0000 7f315ffe6700 -1 reset not still connected to 0x55c1f5890dd0 2024-09-16T12:31:53.541 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:31:53.529+0000 7f315ffe6700 -1 reset not still connected to 0x55c1fae992b0 2024-09-16T12:31:53.541 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:31:53.529+0000 7f315ffe6700 -1 reset not still connected to 0x55c1fae99520 2024-09-16T12:31:53.541 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:31:53.529+0000 7f315ffe6700 -1 reset not still connected to 0x55c1fb4556c0 2024-09-16T12:31:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:31:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:31:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:31:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:31:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:31:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:31:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 764 2024-09-16T12:31:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 762 2024-09-16T12:31:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:31:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:31:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:31:54.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 768 2024-09-16T12:31:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:31:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:31:54.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:31:54.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:31:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:31:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:31:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:31:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:31:55.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:31:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:31:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:31:55.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:31:55.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 770 2024-09-16T12:31:55.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 771 2024-09-16T12:31:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:55.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:31:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:31:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:31:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:31:56.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:31:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:31:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 775 2024-09-16T12:31:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 774 2024-09-16T12:31:56.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:31:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:57.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:57.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:31:57.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:31:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 778 2024-09-16T12:31:57.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 776 2024-09-16T12:31:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:57.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 777 2024-09-16T12:31:57.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:31:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:31:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:31:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:31:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:31:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:31:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 785 2024-09-16T12:31:58.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 783 2024-09-16T12:31:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:31:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:31:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:31:59.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:31:59.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:31:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:31:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:31:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:31:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:31:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:32:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 788 2024-09-16T12:32:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:00.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:32:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:32:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:32:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:32:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:32:00.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:32:00.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 790 2024-09-16T12:32:00.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 789 2024-09-16T12:32:00.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:00.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:32:00.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:00.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:00.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:32:00.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:32:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:00.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:00.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:32:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:32:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:32:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:32:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:32:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:32:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 794 2024-09-16T12:32:01.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 795 2024-09-16T12:32:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:01.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 798 2024-09-16T12:32:02.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 799 2024-09-16T12:32:02.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 800 2024-09-16T12:32:02.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:02.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:02.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 803 2024-09-16T12:32:03.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:03.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 805 2024-09-16T12:32:03.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:32:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:32:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:32:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:32:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:32:03.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:32:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:32:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:32:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:32:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:32:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:32:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:32:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:32:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:32:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:32:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 809 2024-09-16T12:32:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:32:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:32:05.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:32:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 808 2024-09-16T12:32:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:32:05.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:32:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:32:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 810 2024-09-16T12:32:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:32:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:32:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:32:05.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:05.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:32:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:32:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:32:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:32:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:32:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:32:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 814 2024-09-16T12:32:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 815 2024-09-16T12:32:06.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:06.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 818 2024-09-16T12:32:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 817 2024-09-16T12:32:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 820 2024-09-16T12:32:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:08.698 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:32:08.688+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T12:32:08.708 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:32:08.692+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T12:32:08.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 825 2024-09-16T12:32:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:08.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:32:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:32:08.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:32:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:08.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 822 2024-09-16T12:32:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:32:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:32:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:32:08.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:32:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:32:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:32:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:32:09.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:32:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:32:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:32:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:32:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:32:10.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 829 2024-09-16T12:32:10.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 828 2024-09-16T12:32:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:32:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:32:10.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:32:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:32:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:32:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:32:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:32:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:32:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:32:10.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:10.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 830 2024-09-16T12:32:10.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:11.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 835 2024-09-16T12:32:11.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:11.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:32:11.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:11.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:32:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:32:11.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 834 2024-09-16T12:32:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:32:11.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:32:11.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:32:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:11.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:12.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 840 2024-09-16T12:32:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 839 2024-09-16T12:32:12.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 838 2024-09-16T12:32:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:13.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 844 2024-09-16T12:32:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:32:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:32:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:32:13.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 842 2024-09-16T12:32:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:32:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:32:13.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:32:13.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:13.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:13.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:32:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:32:13.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:32:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:32:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:32:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:32:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:32:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:32:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:32:15.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:32:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:32:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:32:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:32:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:32:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:32:15.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:32:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:32:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:32:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 848 2024-09-16T12:32:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 850 2024-09-16T12:32:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 849 2024-09-16T12:32:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:15.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:16.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:32:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:32:16.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:32:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:32:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:16.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:32:16.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:32:16.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 854 2024-09-16T12:32:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 855 2024-09-16T12:32:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:16.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 858 2024-09-16T12:32:17.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:17.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 856 2024-09-16T12:32:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 857 2024-09-16T12:32:17.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:17.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:18.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 865 2024-09-16T12:32:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 864 2024-09-16T12:32:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:32:18.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:32:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:32:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:32:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:18.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:32:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:32:18.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:18.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:18.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:32:18.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:18.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:18.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:32:18.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:32:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:32:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:32:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:32:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:32:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:32:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:32:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:32:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:20.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:32:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:32:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 870 2024-09-16T12:32:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 869 2024-09-16T12:32:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:32:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:32:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:32:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:32:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:32:20.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:32:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 868 2024-09-16T12:32:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:32:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:21.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:32:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:32:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:32:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:32:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:32:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 874 2024-09-16T12:32:21.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 875 2024-09-16T12:32:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:21.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:22.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 877 2024-09-16T12:32:22.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 879 2024-09-16T12:32:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 880 2024-09-16T12:32:22.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:23.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 884 2024-09-16T12:32:23.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 885 2024-09-16T12:32:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:32:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:32:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:32:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:32:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:32:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:32:23.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:32:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:32:23.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:32:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:32:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:32:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:32:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:24.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:32:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:32:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:32:25.446 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:32:25.436+0000 7f61b1855700 -1 reset not still connected to 0x557b3caf24e0 2024-09-16T12:32:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:32:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:32:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:32:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:32:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:32:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:32:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:32:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:32:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:32:25.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 888 2024-09-16T12:32:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 889 2024-09-16T12:32:25.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:25.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 890 2024-09-16T12:32:25.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:32:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:32:26.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:32:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:32:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:26.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:32:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:32:26.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 894 2024-09-16T12:32:26.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 895 2024-09-16T12:32:26.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 897 2024-09-16T12:32:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 898 2024-09-16T12:32:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 896 2024-09-16T12:32:27.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:27.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 905 2024-09-16T12:32:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:32:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:32:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:32:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:32:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:32:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:32:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:32:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:32:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:32:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 904 2024-09-16T12:32:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:32:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:32:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:32:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:32:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:32:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:32:30.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:32:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:32:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:32:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:32:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:32:30.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:32:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:32:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:32:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:32:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 909 2024-09-16T12:32:30.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 910 2024-09-16T12:32:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 908 2024-09-16T12:32:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:30.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:32:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:32:31.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:32:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:32:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:32:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:32:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 915 2024-09-16T12:32:31.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 914 2024-09-16T12:32:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:31.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:32.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 918 2024-09-16T12:32:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 920 2024-09-16T12:32:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 919 2024-09-16T12:32:32.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:32.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:33.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 924 2024-09-16T12:32:33.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 925 2024-09-16T12:32:33.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:33.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:33.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:32:33.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:33.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:33.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:32:33.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:32:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:32:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:32:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:32:33.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:33.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:32:33.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:33.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:32:33.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:32:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:32:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:32:34.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:32:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:32:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:32:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:32:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 929 2024-09-16T12:32:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 930 2024-09-16T12:32:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:32:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:32:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:32:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:35.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:32:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:32:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:32:35.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 928 2024-09-16T12:32:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:32:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:32:35.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:32:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:36.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:32:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:32:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:32:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:32:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:32:36.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:32:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 935 2024-09-16T12:32:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 934 2024-09-16T12:32:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 940 2024-09-16T12:32:37.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 939 2024-09-16T12:32:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 938 2024-09-16T12:32:37.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 944 2024-09-16T12:32:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:32:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:32:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:32:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:32:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:32:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:32:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 945 2024-09-16T12:32:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:32:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:32:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:32:38.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:32:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:32:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:32:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:32:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:32:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:32:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:32:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:32:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:32:40.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 948 2024-09-16T12:32:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:32:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:32:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:32:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 949 2024-09-16T12:32:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:32:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:32:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:32:40.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 950 2024-09-16T12:32:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:40.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:40.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:32:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:32:41.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:32:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 955 2024-09-16T12:32:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:32:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:32:41.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:32:41.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 954 2024-09-16T12:32:41.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:41.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:42.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 960 2024-09-16T12:32:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:42.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 959 2024-09-16T12:32:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 957 2024-09-16T12:32:42.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:42.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 964 2024-09-16T12:32:43.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 965 2024-09-16T12:32:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:32:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:43.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:32:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:32:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:32:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:32:43.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:32:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:32:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:32:43.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:32:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:32:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:32:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:32:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:32:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:32:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:32:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:32:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:32:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:32:45.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:32:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:32:45.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:32:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:32:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:32:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:32:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 969 2024-09-16T12:32:45.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 970 2024-09-16T12:32:45.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 968 2024-09-16T12:32:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:45.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:32:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:32:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:32:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:32:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:32:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:32:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 974 2024-09-16T12:32:46.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 973 2024-09-16T12:32:46.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:47.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 978 2024-09-16T12:32:47.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 977 2024-09-16T12:32:47.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 976 2024-09-16T12:32:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 985 2024-09-16T12:32:48.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 984 2024-09-16T12:32:48.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:48.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:32:48.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:48.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:48.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:32:48.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:32:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:32:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:48.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:32:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:32:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:32:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:32:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:32:48.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:32:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:32:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:32:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:49.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:32:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:32:49.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:32:50.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 988 2024-09-16T12:32:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 989 2024-09-16T12:32:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 990 2024-09-16T12:32:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:32:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:32:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:32:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:32:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:32:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:32:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:32:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:32:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:32:50.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:50.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:32:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:32:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:32:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:32:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:32:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:32:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 994 2024-09-16T12:32:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 995 2024-09-16T12:32:51.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:51.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:52.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1000 2024-09-16T12:32:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 998 2024-09-16T12:32:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:52.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:52.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 999 2024-09-16T12:32:52.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:53.542 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:32:53.532+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e36ab1e0 2024-09-16T12:32:53.543 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:32:53.532+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e5d9cf70 2024-09-16T12:32:53.544 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:32:53.532+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e5d9d2b0 2024-09-16T12:32:53.544 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:32:53.532+0000 7f315ffe6700 -1 reset not still connected to 0x55c1f5890dd0 2024-09-16T12:32:53.544 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:32:53.532+0000 7f315ffe6700 -1 reset not still connected to 0x55c1fae99520 2024-09-16T12:32:53.544 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:32:53.532+0000 7f315ffe6700 -1 reset not still connected to 0x55c1fb4556c0 2024-09-16T12:32:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1004 2024-09-16T12:32:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1005 2024-09-16T12:32:53.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:53.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:32:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:32:53.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:32:53.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:32:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:32:53.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:32:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:32:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:32:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:32:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:32:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:32:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:32:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:32:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:32:54.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:32:55.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1008 2024-09-16T12:32:55.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1009 2024-09-16T12:32:55.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1010 2024-09-16T12:32:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:32:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:32:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:32:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:32:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:32:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:32:55.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:55.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:55.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:32:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:32:55.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:32:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:32:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:32:56.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:32:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:32:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:32:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:32:56.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1015 2024-09-16T12:32:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1014 2024-09-16T12:32:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:56.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1020 2024-09-16T12:32:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1019 2024-09-16T12:32:57.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1018 2024-09-16T12:32:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:32:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:32:58.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:32:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1024 2024-09-16T12:32:58.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1025 2024-09-16T12:32:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:32:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:32:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:32:58.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:32:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:32:58.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:32:58.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:58.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:32:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:32:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:32:59.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:32:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:32:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:32:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:32:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:32:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:32:59.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:33:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1028 2024-09-16T12:33:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:00.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:33:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:33:00.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:33:00.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:00.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:33:00.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:00.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:00.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:33:00.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:33:00.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:33:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1030 2024-09-16T12:33:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:33:00.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:33:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1029 2024-09-16T12:33:00.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:00.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:33:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:33:01.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:33:01.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1035 2024-09-16T12:33:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:33:01.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:33:01.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:33:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1034 2024-09-16T12:33:01.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:01.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:02.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1040 2024-09-16T12:33:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1039 2024-09-16T12:33:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1038 2024-09-16T12:33:02.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:02.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:03.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1045 2024-09-16T12:33:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:33:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:03.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:33:03.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:33:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1044 2024-09-16T12:33:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:33:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:33:03.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:33:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:33:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:33:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:33:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:04.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:33:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:33:04.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:33:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:04.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:33:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:33:04.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:33:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:33:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:05.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:33:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:33:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:33:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:33:05.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:33:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:33:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:33:05.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:33:05.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1050 2024-09-16T12:33:05.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1048 2024-09-16T12:33:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1049 2024-09-16T12:33:05.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:05.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:33:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:33:06.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:33:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:33:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:06.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:33:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:33:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1054 2024-09-16T12:33:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1055 2024-09-16T12:33:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:06.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1058 2024-09-16T12:33:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1057 2024-09-16T12:33:07.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:07.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1056 2024-09-16T12:33:07.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:08.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1064 2024-09-16T12:33:08.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1065 2024-09-16T12:33:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:33:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:33:08.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:33:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:33:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:33:08.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:33:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:33:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:33:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:33:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:33:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:33:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:33:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:33:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:33:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:33:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:33:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:33:10.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:33:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:33:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:33:10.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:33:10.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1068 2024-09-16T12:33:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:33:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:33:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:33:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1070 2024-09-16T12:33:10.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:10.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1069 2024-09-16T12:33:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:33:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:33:11.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:33:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:33:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:33:11.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:33:11.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1074 2024-09-16T12:33:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1075 2024-09-16T12:33:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:11.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1079 2024-09-16T12:33:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1080 2024-09-16T12:33:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1076 2024-09-16T12:33:12.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1085 2024-09-16T12:33:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1084 2024-09-16T12:33:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:33:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:33:13.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:33:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:33:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:33:13.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:33:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:33:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:33:13.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:33:14.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:14.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:33:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:33:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:33:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:33:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:33:14.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:33:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:33:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:33:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:33:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:33:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:15.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:33:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:33:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:33:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:15.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:33:15.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:33:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1088 2024-09-16T12:33:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:15.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1089 2024-09-16T12:33:15.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1090 2024-09-16T12:33:15.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:15.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:16.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1095 2024-09-16T12:33:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:33:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:33:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:33:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1094 2024-09-16T12:33:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:33:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:33:16.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:33:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:16.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:17.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1098 2024-09-16T12:33:17.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1097 2024-09-16T12:33:17.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1096 2024-09-16T12:33:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:18.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1105 2024-09-16T12:33:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:33:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:33:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:33:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1104 2024-09-16T12:33:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:18.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:33:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:33:18.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:33:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:33:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:33:18.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:33:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:33:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:33:19.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:33:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:33:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:33:19.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:33:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:33:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:20.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:33:20.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:33:20.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:33:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:33:20.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:33:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1108 2024-09-16T12:33:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:33:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:33:20.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:33:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1110 2024-09-16T12:33:20.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1109 2024-09-16T12:33:20.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:33:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:33:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:33:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:21.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:33:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:33:21.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:33:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1115 2024-09-16T12:33:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1114 2024-09-16T12:33:21.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1119 2024-09-16T12:33:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1120 2024-09-16T12:33:22.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1118 2024-09-16T12:33:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:22.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:22.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1125 2024-09-16T12:33:23.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1124 2024-09-16T12:33:23.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:23.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:33:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:33:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:33:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:33:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:33:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:33:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:33:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:33:23.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:33:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:33:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:33:24.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:33:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:33:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:24.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:33:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:33:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1128 2024-09-16T12:33:25.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1129 2024-09-16T12:33:25.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1130 2024-09-16T12:33:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:25.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:25.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:25.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:33:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:33:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:33:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:33:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:33:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:33:25.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:33:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:33:25.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:33:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:33:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:33:26.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:33:26.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1134 2024-09-16T12:33:26.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1135 2024-09-16T12:33:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:33:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:33:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:33:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:26.769 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:33:26.755+0000 7f61b1855700 -1 reset not still connected to 0x557b38fa04e0 2024-09-16T12:33:26.770 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:33:26.755+0000 7f61b1855700 -1 reset not still connected to 0x557b3caf36c0 2024-09-16T12:33:26.770 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:33:26.755+0000 7f61b1855700 -1 reset not still connected to 0x557b41942820 2024-09-16T12:33:26.770 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:33:26.755+0000 7f61b1855700 -1 reset not still connected to 0x557b41a5a750 2024-09-16T12:33:27.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1138 2024-09-16T12:33:27.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1139 2024-09-16T12:33:27.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1140 2024-09-16T12:33:27.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:27.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:33:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:33:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:33:28.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1145 2024-09-16T12:33:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:33:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:33:28.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:33:28.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1144 2024-09-16T12:33:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:33:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:33:28.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:33:28.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:28.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:33:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:33:29.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:33:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:33:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:33:29.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:33:30.759 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:33:30.747+0000 7f61b1855700 -1 reset not still connected to 0x557b38fa1ba0 2024-09-16T12:33:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:33:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:33:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:33:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1150 2024-09-16T12:33:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:33:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:33:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:33:30.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1148 2024-09-16T12:33:30.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1149 2024-09-16T12:33:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:30.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:33:30.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:30.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:30.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:33:30.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:33:30.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:30.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:33:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:33:31.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:33:31.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1155 2024-09-16T12:33:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:33:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:33:31.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:33:31.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1154 2024-09-16T12:33:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:31.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1158 2024-09-16T12:33:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1160 2024-09-16T12:33:32.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1159 2024-09-16T12:33:32.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:32.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:32.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:33.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1164 2024-09-16T12:33:33.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1165 2024-09-16T12:33:33.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:33.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:33:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:33:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:33:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:33:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:33:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:33:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:33:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:33:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:33:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:33:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:33:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:33:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:33:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:33:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:33:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:33:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:33:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:33:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:33:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:33:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:33:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:33:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:33:35.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:33:35.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1169 2024-09-16T12:33:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1168 2024-09-16T12:33:35.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:35.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1170 2024-09-16T12:33:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:36.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:33:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:33:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:33:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:33:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:33:36.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:33:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1174 2024-09-16T12:33:36.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1175 2024-09-16T12:33:36.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:36.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1180 2024-09-16T12:33:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1178 2024-09-16T12:33:37.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1177 2024-09-16T12:33:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:37.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1184 2024-09-16T12:33:38.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:38.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1185 2024-09-16T12:33:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:33:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:33:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:33:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:33:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:33:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:33:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:33:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:33:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:33:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:39.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:33:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:33:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:33:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:33:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:33:39.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:33:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:33:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:33:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:33:40.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1190 2024-09-16T12:33:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:33:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:33:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:33:40.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1188 2024-09-16T12:33:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1189 2024-09-16T12:33:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:33:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:33:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:33:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:40.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:41.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:33:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:33:41.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:33:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1194 2024-09-16T12:33:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1195 2024-09-16T12:33:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:33:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:33:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:33:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:41.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1200 2024-09-16T12:33:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1199 2024-09-16T12:33:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1198 2024-09-16T12:33:42.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:43.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1204 2024-09-16T12:33:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1205 2024-09-16T12:33:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:33:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:33:43.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:33:43.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:43.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:33:43.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:33:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:33:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:33:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:33:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:33:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:33:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:33:44.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:33:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:33:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:33:44.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:33:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:33:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:33:45.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:33:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:33:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:45.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:33:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:33:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:33:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:33:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:33:45.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1209 2024-09-16T12:33:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1208 2024-09-16T12:33:45.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:45.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1210 2024-09-16T12:33:45.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:45.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:46.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:33:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:33:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:33:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:33:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:33:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:33:46.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1214 2024-09-16T12:33:46.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1215 2024-09-16T12:33:46.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:46.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:47.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1216 2024-09-16T12:33:47.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1218 2024-09-16T12:33:47.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1217 2024-09-16T12:33:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:48.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1224 2024-09-16T12:33:48.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1225 2024-09-16T12:33:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:33:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:33:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:33:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:48.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:33:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:33:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:33:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:33:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:33:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:33:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:33:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:33:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:33:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:33:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:33:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:33:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:33:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:33:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:33:50.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1229 2024-09-16T12:33:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:33:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:33:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:33:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:33:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:33:50.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:33:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1228 2024-09-16T12:33:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1230 2024-09-16T12:33:50.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:51.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:33:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:33:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:33:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:33:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:33:51.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:33:51.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1234 2024-09-16T12:33:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1235 2024-09-16T12:33:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:52.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1239 2024-09-16T12:33:52.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1240 2024-09-16T12:33:52.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1237 2024-09-16T12:33:52.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:52.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:52.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:53.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1244 2024-09-16T12:33:53.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1245 2024-09-16T12:33:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:53.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:33:53.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 39419 2024-09-16T12:33:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:33:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:33:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:33:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:33:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:53.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:33:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:33:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:54.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:33:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:33:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:33:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:33:54.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 39553 2024-09-16T12:33:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1249 2024-09-16T12:33:55.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:55.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1248 2024-09-16T12:33:55.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:55.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:33:55.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:55.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:55.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:33:55.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:33:55.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:33:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:33:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:33:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:33:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 39687 2024-09-16T12:33:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1252 2024-09-16T12:33:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:56.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:33:56.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 39841 2024-09-16T12:33:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:56.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:33:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:33:56.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:33:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1254 2024-09-16T12:33:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1255 2024-09-16T12:33:57.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:58.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1258 2024-09-16T12:33:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:33:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:33:58.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:33:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:33:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:33:58.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:33:58.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:33:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:33:58.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 40029 2024-09-16T12:33:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:33:59.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 40184 2024-09-16T12:34:00.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1260 2024-09-16T12:34:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:34:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 40319 2024-09-16T12:34:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:34:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:34:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:34:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:34:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:34:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:34:00.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:34:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:34:01.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 40494 2024-09-16T12:34:02.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1262 2024-09-16T12:34:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:34:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:34:03.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 40740 2024-09-16T12:34:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T12:34:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:34:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 39286 2024-09-16T12:36:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:36:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:36:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:36:25.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 39420 2024-09-16T12:36:26.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:36:26.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:36:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:36:26.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 39554 2024-09-16T12:36:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:36:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:36:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:36:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 39688 2024-09-16T12:36:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:36:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:36:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:36:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 39842 2024-09-16T12:36:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:36:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:36:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:36:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 40030 2024-09-16T12:36:31.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:36:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:36:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:36:31.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 40185 2024-09-16T12:36:32.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:36:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:36:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:36:32.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 40320 2024-09-16T12:36:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:36:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:36:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:36:33.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 40495 2024-09-16T12:36:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:36:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:36:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:36:34.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 40741 2024-09-16T12:36:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:36:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:36:36.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T12:36:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:36:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:36:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:36:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 1 2024-09-16T12:36:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:36:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:36:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd0 2024-09-16T12:36:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 2 2024-09-16T12:36:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary1 2024-09-16T12:36:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary1 2024-09-16T12:36:36.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:36:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary1 2024-09-16T12:36:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:36:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd1 2024-09-16T12:36:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary2 2024-09-16T12:36:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary2 2024-09-16T12:36:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:36:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:36:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary2 2024-09-16T12:36:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:36:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 3 2024-09-16T12:36:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:36:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 4 2024-09-16T12:36:36.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd2 2024-09-16T12:36:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary3 2024-09-16T12:36:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary3 2024-09-16T12:36:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:36:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary3 2024-09-16T12:36:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:36:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd3 2024-09-16T12:36:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:36:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary4 2024-09-16T12:36:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary4 2024-09-16T12:36:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:36:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary4 2024-09-16T12:36:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 5 2024-09-16T12:36:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:36:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:36:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 6 2024-09-16T12:36:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd4 2024-09-16T12:36:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary5 2024-09-16T12:36:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary5 2024-09-16T12:36:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:36:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:36:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary5 2024-09-16T12:36:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:36:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd5 2024-09-16T12:36:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary6 2024-09-16T12:36:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary6 2024-09-16T12:36:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:36:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary6 2024-09-16T12:36:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:36:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:36:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 8 2024-09-16T12:36:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:36:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:36:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd7 2024-09-16T12:36:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary8 2024-09-16T12:36:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary8 2024-09-16T12:36:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:36:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary8 2024-09-16T12:36:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 7 2024-09-16T12:36:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd6 2024-09-16T12:36:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary7 2024-09-16T12:36:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:36:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 9 2024-09-16T12:36:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary7 2024-09-16T12:36:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:36:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary7 2024-09-16T12:36:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:36:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 46711 2024-09-16T12:36:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd8 2024-09-16T12:36:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary9 2024-09-16T12:36:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary9 2024-09-16T12:36:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 10 2024-09-16T12:36:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:36:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary9 2024-09-16T12:36:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd9 2024-09-16T12:36:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary10 2024-09-16T12:36:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary10 2024-09-16T12:36:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:36:36.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary10 2024-09-16T12:36:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd5 2024-09-16T12:36:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:36:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd2 2024-09-16T12:36:37.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:36:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd1 2024-09-16T12:36:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:36:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd0 2024-09-16T12:36:37.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:36:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd4 2024-09-16T12:36:37.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:36:37.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:36:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd6 2024-09-16T12:36:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:36:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd9 2024-09-16T12:36:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:36:37.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd3 2024-09-16T12:36:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:36:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd7 2024-09-16T12:36:37.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:36:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd8 2024-09-16T12:37:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=05aae8dc9ad5c02b275ed1ab43c418d9 2024-09-16T12:37:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary3 2024-09-16T12:37:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=f014717d9f55ff49f3f7a6e8cd1d608d 2024-09-16T12:37:32.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary6 2024-09-16T12:37:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=ed9317b1899d45698debab78fc5429c4 2024-09-16T12:37:33.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary9 2024-09-16T12:37:33.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=634c6ac62d93a46a3f23d9f43e31b018 2024-09-16T12:37:33.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary10 2024-09-16T12:37:33.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-16T12:37:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:37:33.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-16T12:37:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-16T12:37:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:37:33.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-16T12:37:34.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=ce9e85be395180b955c4b72e6f5f797b 2024-09-16T12:37:34.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary7 2024-09-16T12:37:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=494667b39c27401bcac1e35a19dedc0d 2024-09-16T12:37:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary8 2024-09-16T12:37:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=fbb1972b29766b8875f5c7f49a914a99 2024-09-16T12:37:34.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary4 2024-09-16T12:37:34.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=82bbeb220348704a68960e08ec44f1fb 2024-09-16T12:37:34.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary2 2024-09-16T12:37:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-16T12:37:34.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:37:34.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-16T12:37:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=5a54ef8e9bf7033dd79902943d1f3799 2024-09-16T12:37:34.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary5 2024-09-16T12:37:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-16T12:37:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:37:34.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-16T12:37:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-16T12:37:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:34.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:37:34.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-16T12:37:34.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-16T12:37:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:37:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-16T12:37:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-16T12:37:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:37:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-16T12:37:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=469d4d960720eea0b0e184bb6744473f 2024-09-16T12:37:34.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary1 2024-09-16T12:37:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-16T12:37:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:37:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-16T12:37:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-16T12:37:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:37:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-16T12:37:35.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:37:35.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:37:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-16T12:37:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:37:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:37:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-16T12:37:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:37:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:37:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-16T12:37:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:35.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:37:35.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-16T12:37:36.251 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:37:36.237+0000 7f74def64700 -1 reset not still connected to 0x559441c00a90 2024-09-16T12:37:36.251 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:37:36.237+0000 7f74def64700 -1 reset not still connected to 0x559453914820 2024-09-16T12:37:36.252 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:37:36.237+0000 7f74def64700 -1 reset not still connected to 0x559453915110 2024-09-16T12:37:36.252 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:37:36.237+0000 7f74def64700 -1 reset not still connected to 0x559441c00a90 2024-09-16T12:37:36.252 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:37:36.237+0000 7f74def64700 -1 reset not still connected to 0x559453915110 2024-09-16T12:37:36.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:37:36.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:37:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-16T12:37:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:36.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:37:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:36.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:37:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-16T12:37:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-16T12:37:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:37:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:37:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:36.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T12:37:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:37:36.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:37:36.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:37:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:36.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T12:37:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-16T12:37:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:37:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:36.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:37:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-16T12:37:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-16T12:37:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:37:36.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:37:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:37:36.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:37:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-16T12:37:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:37:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:36.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:37:36.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:37:36.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:37:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-16T12:37:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:37:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:37:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-16T12:37:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:37:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:37:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T12:37:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 2d9ab414-6b90-42a5-a42c-00a9400d330b 2024-09-16T12:37:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:37:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:37:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:33 2024-09-16T12:37:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:37:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:36' 2024-09-16T12:37:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T12:37:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 1166f0a7-946b-4553-bdb8-54616925dac3 2024-09-16T12:37:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:37:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:37:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:33 2024-09-16T12:37:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:37:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:36' 2024-09-16T12:37:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:37:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:37:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:37:36.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:37:37.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:37:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-16T12:37:37.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:37:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:37:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-16T12:37:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:37:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:37.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:37.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:37.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T12:37:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-16T12:37:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:37.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:37:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:37.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T12:37:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-16T12:37:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:37:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:37.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T12:37:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-16T12:37:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:37:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-16T12:37:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:37:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T12:37:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-16T12:37:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-16T12:37:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:37:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:37:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:37.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:37.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T12:37:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T12:37:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:37.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T12:37:37.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T12:37:37.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 513096d2-ff05-4526-8ac4-746ec0e8d36c 2024-09-16T12:37:37.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:37:37.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:37:37.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:37.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:33 2024-09-16T12:37:37.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:37.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:37.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:37:37.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":956416.0,"last_snapshot_bytes":184320,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490030,"remote_snapshot_timestamp":1726490030,"replay_state":"idle"} 2024-09-16T12:37:37.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:23' 2024-09-16T12:37:37.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:37:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:37.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:37:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T12:37:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 03a4f57d-6ac5-4d45-8468-694f291bfc5a 2024-09-16T12:37:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:37:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:37:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:33 2024-09-16T12:37:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:37:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2140160.0,"last_snapshot_bytes":2609152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490037,"remote_snapshot_timestamp":1726490037,"replay_state":"idle"} 2024-09-16T12:37:37.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:23' 2024-09-16T12:37:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:37.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:37:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T12:37:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 7c22ff11-90eb-4a64-9753-ee0488c7c6f0 2024-09-16T12:37:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:37:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:37:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:33 2024-09-16T12:37:37.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:37:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:37' 2024-09-16T12:37:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:37:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:37:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:37.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:37:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T12:37:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: f67b4d08-9215-4c3a-be5a-c06ea7bbbb73 2024-09-16T12:37:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:37:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:37:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:33 2024-09-16T12:37:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:37:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2168832.0,"last_snapshot_bytes":2600960,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490041,"remote_snapshot_timestamp":1726490041,"replay_state":"idle"} 2024-09-16T12:37:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:23' 2024-09-16T12:37:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T12:37:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 6588a7f2-9bf0-4c8d-9a72-4664809bda6b 2024-09-16T12:37:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:37:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:37:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:33 2024-09-16T12:37:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:37:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2117632.0,"last_snapshot_bytes":2605056,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490039,"remote_snapshot_timestamp":1726490039,"replay_state":"idle"} 2024-09-16T12:37:37.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:23' 2024-09-16T12:37:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:37:37.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:37.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:37:37.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:37:37.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:37.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:37:37.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T12:37:37.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b2fd01b8-9e0f-4858-992e-38984ef0b6a3 2024-09-16T12:37:37.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:37:37.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:37:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:33 2024-09-16T12:37:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:37:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2156544.0,"last_snapshot_bytes":2584576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490034,"remote_snapshot_timestamp":1726490034,"replay_state":"idle"} 2024-09-16T12:37:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:23' 2024-09-16T12:37:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:37:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:37:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T12:37:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 84b20afe-0f64-4172-b43d-9f6a37443624 2024-09-16T12:37:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:37:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:37:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:33 2024-09-16T12:37:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:37:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":970752.0,"last_snapshot_bytes":1753088,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490036,"remote_snapshot_timestamp":1726490036,"replay_state":"idle"} 2024-09-16T12:37:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:23' 2024-09-16T12:37:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:37:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:37.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:37:38.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-16T12:37:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:37:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:38.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T12:37:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T12:37:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: dba8131a-5e5e-4a91-bc7c-b81e67b290cd 2024-09-16T12:37:38.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:37:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:37:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:33 2024-09-16T12:37:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:37:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38' 2024-09-16T12:37:38.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:37:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:37:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-16T12:37:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-16T12:37:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:37:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:37:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T12:37:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T12:37:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T12:37:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 2d9ab414-6b90-42a5-a42c-00a9400d330b 2024-09-16T12:37:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:37:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:37:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:33 2024-09-16T12:37:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:37:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:36' 2024-09-16T12:37:38.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:37:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 4 2024-09-16T12:37:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T12:37:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 1166f0a7-946b-4553-bdb8-54616925dac3 2024-09-16T12:37:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:37:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:37:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:33 2024-09-16T12:37:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:38.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:37:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:36' 2024-09-16T12:37:38.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:37:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 4 2024-09-16T12:37:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-16T12:37:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:37:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T12:37:39.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-16T12:37:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:37:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:39.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T12:37:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-16T12:37:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:37:39.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:39.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T12:37:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-16T12:37:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:37:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:39.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:39.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T12:37:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-16T12:37:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:37:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:39.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T12:37:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-16T12:37:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:37:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:39.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:39.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T12:37:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-16T12:37:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:37:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:39.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T12:37:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T12:37:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 513096d2-ff05-4526-8ac4-746ec0e8d36c 2024-09-16T12:37:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:39.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38 2024-09-16T12:37:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38' 2024-09-16T12:37:39.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:37:39.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:37:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:37:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:37:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:39.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:37:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:37:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:37:39.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:37:39.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:37:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-16 12:37:38' 2024-09-16T12:37:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:37:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:37:38' 2024-09-16T12:37:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:37:39.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T12:37:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 7c22ff11-90eb-4a64-9753-ee0488c7c6f0 2024-09-16T12:37:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38 2024-09-16T12:37:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38' 2024-09-16T12:37:39.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:37:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:37:39.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:37:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:37:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:37:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:37:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T12:37:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 03a4f57d-6ac5-4d45-8468-694f291bfc5a 2024-09-16T12:37:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:39.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38 2024-09-16T12:37:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T12:37:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b2fd01b8-9e0f-4858-992e-38984ef0b6a3 2024-09-16T12:37:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38 2024-09-16T12:37:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38' 2024-09-16T12:37:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38' 2024-09-16T12:37:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:37:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:37:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:37:39.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:37:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:37:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:37:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:37:39.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:37:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:37:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T12:37:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: f67b4d08-9215-4c3a-be5a-c06ea7bbbb73 2024-09-16T12:37:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38 2024-09-16T12:37:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:39.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38' 2024-09-16T12:37:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:37:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T12:37:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 84b20afe-0f64-4172-b43d-9f6a37443624 2024-09-16T12:37:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38 2024-09-16T12:37:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:39.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38' 2024-09-16T12:37:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:37:39.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:37:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:37:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:37:39.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:37:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:37:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:37:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:37:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:37:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:37:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:37:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:37:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T12:37:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 6588a7f2-9bf0-4c8d-9a72-4664809bda6b 2024-09-16T12:37:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38 2024-09-16T12:37:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:39.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:39.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38' 2024-09-16T12:37:39.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:37:39.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:39.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:39.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:37:39.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:37:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:37:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:37:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:37:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:37:39.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:37:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:37:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:37:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:37:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:37:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-16 12:37:38' 2024-09-16T12:37:39.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:37:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:37:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:37:38' 2024-09-16T12:37:39.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:37:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:37:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-16 12:37:38' 2024-09-16T12:37:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:37:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:37:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:37:38' 2024-09-16T12:37:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:39.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:37:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:37:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-16 12:37:38' 2024-09-16T12:37:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:37:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:37:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:37:38' 2024-09-16T12:37:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:39.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:37:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:37:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:37:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:37:39.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:37:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:37:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:39.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:37:39.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:37:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:37:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:37:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-16 12:37:38' 2024-09-16T12:37:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:37:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:37:38' 2024-09-16T12:37:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:37:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-16 12:37:38' 2024-09-16T12:37:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:37:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:37:38' 2024-09-16T12:37:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:37:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-16 12:37:38' 2024-09-16T12:37:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:39.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:37:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:37:38' 2024-09-16T12:37:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:39.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:37:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38' 2024-09-16T12:37:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:37:39.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T12:37:39.871 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:37:39.855+0000 7f61b1855700 -1 reset not still connected to 0x557b382045b0 2024-09-16T12:37:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:39.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T12:37:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:37:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:37:39.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-16T12:37:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:37:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:37:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38' 2024-09-16T12:37:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:37:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T12:37:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38' 2024-09-16T12:37:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:37:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T12:37:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T12:37:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T12:37:39.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:37:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:37:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-16T12:37:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:37:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:39.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:37:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:37:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:37:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2<+ return 0 2024-09-16T12:37:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mirror_uuids>c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38' 2024-09-16T12:37:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:37:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-16T12:37:39.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:37:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:37:39.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T12:37:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38' 2024-09-16T12:37:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:37:39.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T12:37:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38' 2024-09-16T12:37:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:37:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T12:37:39.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:39.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T12:37:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:39.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T12:37:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:37:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:37:39.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T12:37:39.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-16T12:37:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:37:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:39.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:37:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:37:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:37:39.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-16T12:37:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:37:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:37:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:37:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:37:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-16T12:37:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:37:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:39.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:39.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:37:40.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-16T12:37:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:37:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:40.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:40.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T12:37:40.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T12:37:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: dba8131a-5e5e-4a91-bc7c-b81e67b290cd 2024-09-16T12:37:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38 2024-09-16T12:37:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38' 2024-09-16T12:37:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:37:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:37:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:37:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:37:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:40.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:37:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:40.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:37:40.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:37:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:37:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:37:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:40.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-16 12:37:38' 2024-09-16T12:37:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:37:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:37:38' 2024-09-16T12:37:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:37:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38' 2024-09-16T12:37:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:37:40.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T12:37:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T12:37:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:37:40.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:37:40.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-16T12:37:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:37:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:37:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-16T12:37:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:37:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-16T12:37:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38' 2024-09-16T12:37:40.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:37:40.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T12:37:40.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:40.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T12:37:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:37:40.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:37:40.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-16T12:37:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:37:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:40.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:37:40.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-16T12:37:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:37:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:40.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-16T12:37:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-16T12:37:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:37:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-16T12:37:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T12:37:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 513096d2-ff05-4526-8ac4-746ec0e8d36c 2024-09-16T12:37:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38 2024-09-16T12:37:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:37:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38' 2024-09-16T12:37:40.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:37:40.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:37:40.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-16T12:37:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:37:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:40.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:40.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:37:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:37:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-16T12:37:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:37:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-16T12:37:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:37:40.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-16T12:37:40.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:37:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-16T12:37:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:37:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-16T12:37:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:37:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:37:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:37:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-16 12:37:38' 2024-09-16T12:37:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:37:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:37:38' 2024-09-16T12:37:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:37:41.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T12:37:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 6588a7f2-9bf0-4c8d-9a72-4664809bda6b 2024-09-16T12:37:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38 2024-09-16T12:37:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:37:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38' 2024-09-16T12:37:41.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:37:41.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:37:41.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:37:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:37:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:41.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:37:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:41.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:37:41.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:37:41.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:37:41.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:37:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-16 12:37:38' 2024-09-16T12:37:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:37:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:37:38' 2024-09-16T12:37:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:37:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T12:37:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b2fd01b8-9e0f-4858-992e-38984ef0b6a3 2024-09-16T12:37:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38 2024-09-16T12:37:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:37:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38' 2024-09-16T12:37:41.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:37:41.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:37:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T12:37:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 84b20afe-0f64-4172-b43d-9f6a37443624 2024-09-16T12:37:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38 2024-09-16T12:37:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:41.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:37:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38' 2024-09-16T12:37:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:41.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:37:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:37:41.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:37:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T12:37:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 7c22ff11-90eb-4a64-9753-ee0488c7c6f0 2024-09-16T12:37:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:41.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:41.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:41.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38 2024-09-16T12:37:41.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:41.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:37:41.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:41.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:41.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38' 2024-09-16T12:37:41.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:37:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:37:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T12:37:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 03a4f57d-6ac5-4d45-8468-694f291bfc5a 2024-09-16T12:37:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38 2024-09-16T12:37:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:37:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38' 2024-09-16T12:37:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:37:41.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:37:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:37:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:37:41.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:37:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:37:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:37:41.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:37:41.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:37:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:37:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:37:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:37:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:37:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:37:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:37:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:37:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:37:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:41.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:37:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:37:41.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:37:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:37:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:37:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-16 12:37:38' 2024-09-16T12:37:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:37:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:37:38' 2024-09-16T12:37:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:37:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-16 12:37:38' 2024-09-16T12:37:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:37:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:37:38' 2024-09-16T12:37:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:37:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:37:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:37:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:37:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:37:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:37:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:37:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-16 12:37:38' 2024-09-16T12:37:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:37:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:37:38' 2024-09-16T12:37:41.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-16 12:37:38' 2024-09-16T12:37:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:37:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:37:38' 2024-09-16T12:37:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:37:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:37:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:37:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38' 2024-09-16T12:37:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:37:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T12:37:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T12:37:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:37:41.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:37:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:41.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-16T12:37:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:37:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:37:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-16T12:37:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38' 2024-09-16T12:37:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:37:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T12:37:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T12:37:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:37:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:37:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:41.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-16T12:37:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:37:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:37:41.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-16T12:37:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38' 2024-09-16T12:37:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:37:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T12:37:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38' 2024-09-16T12:37:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:37:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T12:37:41.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:41.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T12:37:41.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38' 2024-09-16T12:37:41.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:37:41.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:41.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T12:37:41.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T12:37:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:37:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:37:41.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:41.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38' 2024-09-16T12:37:41.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:37:41.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-16T12:37:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:37:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:37:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-16T12:37:41.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T12:37:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T12:37:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:37:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:37:41.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:41.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-16T12:37:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:37:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:37:41.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-16T12:37:41.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:37:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:37:41.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:41.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-16T12:37:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:37:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:37:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-16T12:37:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T12:37:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:37:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:41.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:37:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:41.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-16T12:37:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:37:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:37:41.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-16T12:37:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-16T12:37:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:37:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:41.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:41.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-16T12:37:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T12:37:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: dba8131a-5e5e-4a91-bc7c-b81e67b290cd 2024-09-16T12:37:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:41.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38 2024-09-16T12:37:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:37:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38' 2024-09-16T12:37:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:41.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:37:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:37:41.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:37:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:37:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:37:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:37:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:37:41.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:37:41.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:37:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:41.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-16 12:37:38' 2024-09-16T12:37:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:41.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:37:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:37:38' 2024-09-16T12:37:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:41.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:37:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38' 2024-09-16T12:37:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:37:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T12:37:41.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-16T12:37:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:37:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-16T12:37:41.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T12:37:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:37:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:41.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:37:41.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-16T12:37:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:37:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:37:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-16T12:37:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T12:37:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: f67b4d08-9215-4c3a-be5a-c06ea7bbbb73 2024-09-16T12:37:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38 2024-09-16T12:37:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:37:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38' 2024-09-16T12:37:42.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:37:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:37:42.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:37:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:37:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:42.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:37:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:42.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:37:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:37:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:37:42.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:37:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-16 12:37:38' 2024-09-16T12:37:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:37:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:37:38' 2024-09-16T12:37:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:42.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:37:42.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:38cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:38' 2024-09-16T12:37:42.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:37:42.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T12:37:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T12:37:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:37:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:42.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:37:42.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-16T12:37:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:42.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:37:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:37:42.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-16T12:37:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:37:42.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:37:42.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:37:42.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:37:42.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:37:42.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:37:42.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:37:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:37:42.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:37:42.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary1 2024-09-16T12:37:42.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:37:42.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:37:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:37:42.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary4 2024-09-16T12:37:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary2 2024-09-16T12:37:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:37:42.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary8 2024-09-16T12:37:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary5 2024-09-16T12:37:42.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:37:42.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary9 2024-09-16T12:37:42.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:37:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary7 2024-09-16T12:37:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:37:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary10 2024-09-16T12:37:42.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd0 2024-09-16T12:37:42.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:37:42.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd0 2024-09-16T12:37:42.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd1 2024-09-16T12:37:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:37:42.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd1 2024-09-16T12:37:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd2 2024-09-16T12:37:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd2 2024-09-16T12:37:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:37:42.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd3 2024-09-16T12:37:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:37:42.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd3 2024-09-16T12:37:42.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd4 2024-09-16T12:37:42.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:37:42.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd4 2024-09-16T12:37:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd5 2024-09-16T12:37:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:37:42.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd5 2024-09-16T12:37:42.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-16T12:37:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:37:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:42.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T12:37:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-16T12:37:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:37:42.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:42.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T12:37:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd6 2024-09-16T12:37:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:37:42.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd6 2024-09-16T12:37:42.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd7 2024-09-16T12:37:42.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:37:42.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd7 2024-09-16T12:37:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T12:37:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 1166f0a7-946b-4553-bdb8-54616925dac3 2024-09-16T12:37:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38 2024-09-16T12:37:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38' 2024-09-16T12:37:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:37:42.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:37:42.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T12:37:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 2d9ab414-6b90-42a5-a42c-00a9400d330b 2024-09-16T12:37:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38 2024-09-16T12:37:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:37:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:38' 2024-09-16T12:37:42.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:37:42.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:37:42.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:42.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:37:42.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:37:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:37:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:42.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:37:42.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:37:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:37:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:37:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:37:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:37:42.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:37:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-16 12:37:38' 2024-09-16T12:37:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:37:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:37:38' 2024-09-16T12:37:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:42.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:37:42.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:42.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:42.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:37:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:37:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:37:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:37:42.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-16 12:37:38' 2024-09-16T12:37:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:37:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:37:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:37:38' 2024-09-16T12:37:42.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:37:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38' 2024-09-16T12:37:43.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:37:43.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T12:37:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:37:38cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:37:38' 2024-09-16T12:37:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:37:43.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:43.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T12:37:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T12:37:43.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:37:43.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:38' 2024-09-16T12:37:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:37:43.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:43.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T12:37:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-16T12:37:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:37:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:37:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:37:43.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:43.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:37:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-16T12:37:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:37:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:43.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:37:43.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:37:44.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-16T12:37:44.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:44.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:44.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:37:44.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:44.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:44.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:44.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-16T12:37:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-16T12:37:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:37:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:37:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:37:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-16T12:37:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T12:37:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 2d9ab414-6b90-42a5-a42c-00a9400d330b 2024-09-16T12:37:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:44.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:43 2024-09-16T12:37:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:37:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:43' 2024-09-16T12:37:44.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:37:44.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:37:44.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:37:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T12:37:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 1166f0a7-946b-4553-bdb8-54616925dac3 2024-09-16T12:37:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:37:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:43 2024-09-16T12:37:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:37:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:37:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:37:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:37:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:37:43' 2024-09-16T12:37:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:37:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:37:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:37:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:37:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:37:44.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:37:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:37:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:37:44.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:37:44.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:37:44.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:37:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:37:44.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:43' 2024-09-16T12:37:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:37:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-16 12:37:43' 2024-09-16T12:37:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:37:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:37:43' 2024-09-16T12:37:44.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:37:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:37:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:37:44.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:37:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:43' 2024-09-16T12:37:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-16 12:37:43' 2024-09-16T12:37:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:37:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:37:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:37:43' 2024-09-16T12:37:44.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:37:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:37:44.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:37:44.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING2842654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}2024-09-16 12:37:43image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}2024-09-16 12:37:43image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}2024-09-16 12:37:43image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:43image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}2024-09-16 12:37:43image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}2024-09-16 12:37:43image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:43image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}2024-09-16 12:37:43image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}2024-09-16 12:37:43image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}2024-09-16 12:37:43' 2024-09-16T12:37:44.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:37:44.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T12:37:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:43' 2024-09-16T12:37:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T12:37:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING2842654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}2024-09-16 12:37:43image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}2024-09-16 12:37:43image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}2024-09-16 12:37:43image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:43image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}2024-09-16 12:37:43image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}2024-09-16 12:37:43image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:37:43image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}2024-09-16 12:37:43image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}2024-09-16 12:37:43image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}2024-09-16 12:37:43' 2024-09-16T12:37:44.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:37:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T12:37:44.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:37:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:37:43' 2024-09-16T12:37:44.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:37:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T12:37:44.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-16T12:37:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:44.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:37:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:37:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-16T12:37:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:37:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:37:44.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:37:44.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:37:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:37:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-16T12:37:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:37:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:37:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:37:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:37:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-16T12:37:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:37:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:37:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary3 2024-09-16T12:37:45.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:37:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:37:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary6 2024-09-16T12:37:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd8 2024-09-16T12:37:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:37:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd8 2024-09-16T12:37:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd9 2024-09-16T12:37:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:37:45.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd9 2024-09-16T12:38:18.392 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:38:18.380+0000 7f74def64700 -1 reset not still connected to 0x5594568d4b60 2024-09-16T12:39:19.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=634c6ac62d93a46a3f23d9f43e31b018 2024-09-16T12:39:19.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd5 2024-09-16T12:39:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=ed9317b1899d45698debab78fc5429c4 2024-09-16T12:39:19.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd4 2024-09-16T12:39:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=469d4d960720eea0b0e184bb6744473f 2024-09-16T12:39:19.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd1 2024-09-16T12:39:19.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=ce9e85be395180b955c4b72e6f5f797b 2024-09-16T12:39:19.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd2 2024-09-16T12:39:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=494667b39c27401bcac1e35a19dedc0d 2024-09-16T12:39:19.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd7 2024-09-16T12:39:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=fbb1972b29766b8875f5c7f49a914a99 2024-09-16T12:39:19.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd3 2024-09-16T12:39:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=5a54ef8e9bf7033dd79902943d1f3799 2024-09-16T12:39:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd6 2024-09-16T12:39:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ ed9317b1899d45698debab78fc5429c4 != \e\d\9\3\1\7\b\1\8\9\9\d\4\5\6\9\8\d\e\b\a\b\7\8\f\c\5\4\2\9\c\4 ]] 2024-09-16T12:39:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 634c6ac62d93a46a3f23d9f43e31b018 != \6\3\4\c\6\a\c\6\2\d\9\3\a\4\6\a\3\f\2\3\d\9\f\4\3\e\3\1\b\0\1\8 ]] 2024-09-16T12:39:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 469d4d960720eea0b0e184bb6744473f != \4\6\9\d\4\d\9\6\0\7\2\0\e\e\a\0\b\0\e\1\8\4\b\b\6\7\4\4\4\7\3\f ]] 2024-09-16T12:39:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:39:20.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 46712 2024-09-16T12:39:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=82bbeb220348704a68960e08ec44f1fb 2024-09-16T12:39:20.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-09-16T12:39:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ ce9e85be395180b955c4b72e6f5f797b != \c\e\9\e\8\5\b\e\3\9\5\1\8\0\b\9\5\5\c\4\b\7\2\e\6\f\5\f\7\9\7\b ]] 2024-09-16T12:39:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 494667b39c27401bcac1e35a19dedc0d != \4\9\4\6\6\7\b\3\9\c\2\7\4\0\1\b\c\a\c\1\e\3\5\a\1\9\d\e\d\c\0\d ]] 2024-09-16T12:39:20.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ fbb1972b29766b8875f5c7f49a914a99 != \f\b\b\1\9\7\2\b\2\9\7\6\6\b\8\8\7\5\f\5\c\7\f\4\9\a\9\1\4\a\9\9 ]] 2024-09-16T12:39:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=f014717d9f55ff49f3f7a6e8cd1d608d 2024-09-16T12:39:20.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd9 2024-09-16T12:39:20.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 5a54ef8e9bf7033dd79902943d1f3799 != \5\a\5\4\e\f\8\e\9\b\f\7\0\3\3\d\d\7\9\9\0\2\9\4\3\d\1\f\3\7\9\9 ]] 2024-09-16T12:39:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=05aae8dc9ad5c02b275ed1ab43c418d9 2024-09-16T12:39:20.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd8 2024-09-16T12:39:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 82bbeb220348704a68960e08ec44f1fb != \8\2\b\b\e\b\2\2\0\3\4\8\7\0\4\a\6\8\9\6\0\e\0\8\e\c\4\4\f\1\f\b ]] 2024-09-16T12:39:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:39:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 46713 2024-09-16T12:39:21.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ f014717d9f55ff49f3f7a6e8cd1d608d != \f\0\1\4\7\1\7\d\9\f\5\5\f\f\4\9\f\3\f\7\a\6\e\8\c\d\1\d\6\0\8\d ]] 2024-09-16T12:39:21.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 05aae8dc9ad5c02b275ed1ab43c418d9 != \0\5\a\a\e\8\d\c\9\a\d\5\c\0\2\b\2\7\5\e\d\1\a\b\4\3\c\4\1\8\d\9 ]] 2024-09-16T12:39:21.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:39:21.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 46716 2024-09-16T12:39:21.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:39:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 46717 2024-09-16T12:39:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:39:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 46719 2024-09-16T12:39:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:39:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 46721 2024-09-16T12:39:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:39:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 46723 2024-09-16T12:39:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:39:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 46725 2024-09-16T12:39:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:39:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 46727 2024-09-16T12:39:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T12:39:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:39:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary1 2024-09-16T12:39:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-16T12:39:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:39:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:21.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:39:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:39:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:39:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:39:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-16T12:39:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:39:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:39:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:39:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:39:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:39:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T12:39:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T12:39:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: dba8131a-5e5e-4a91-bc7c-b81e67b290cd 2024-09-16T12:39:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:39:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70254592.0,"last_snapshot_bytes":70254592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"} 2024-09-16T12:39:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:39:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:38:53 2024-09-16T12:39:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:39:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:39:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:39:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:39:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:39:18' 2024-09-16T12:39:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:39:22.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:39:22.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:39:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70254592.0,"last_snapshot_bytes":70254592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"} 2024-09-16T12:39:22.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:39:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:39:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:39:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:39:22.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:39:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:39:22.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:39:22.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:39:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:38:53' 2024-09-16T12:39:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-16 12:38:53' 2024-09-16T12:39:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:39:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:38:53' 2024-09-16T12:39:22.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:39:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:39:22.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1dba8131a-5e5e-4a91-bc7c-b81e67b290cdup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70254592.0,"last_snapshot_bytes":70254592,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:38:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:18image-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:38:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:18image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72368128.0,"last_snapshot_bytes":72368128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:38:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:18image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72364032.0,"last_snapshot_bytes":72364032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490265,"remote_snapshot_timestamp":1726490265,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:38:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:18image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64479232.0,"last_snapshot_bytes":64479232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:38:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:18image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70144000.0,"last_snapshot_bytes":70144000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:38:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:18image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63856640.0,"last_snapshot_bytes":63856640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490265,"remote_snapshot_timestamp":1726490265,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:38:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:18image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72273920.0,"last_snapshot_bytes":72273920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:38:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:18image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70205440.0,"last_snapshot_bytes":70205440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:38:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:18image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70234112.0,"last_snapshot_bytes":70234112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:38:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:18' 2024-09-16T12:39:22.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:39:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T12:39:22.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:38:53' 2024-09-16T12:39:22.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T12:39:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:39:22.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:39:22.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:39:22.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:39:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-16T12:39:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:39:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:39:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-16T12:39:22.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-16T12:39:24.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:39:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-16T12:39:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:39:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:39:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:39:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary1 2024-09-16T12:39:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary1 2024-09-16T12:39:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:39:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary2 2024-09-16T12:39:24.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-16T12:39:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:39:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:39:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:39:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:39:24.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:39:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-16T12:39:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:39:25.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:39:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:39:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:39:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:39:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T12:39:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T12:39:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 513096d2-ff05-4526-8ac4-746ec0e8d36c 2024-09-16T12:39:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:39:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72368128.0,"last_snapshot_bytes":72368128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"} 2024-09-16T12:39:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:39:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:39:23 2024-09-16T12:39:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:39:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:39:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:39:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:39:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:39:23' 2024-09-16T12:39:25.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:39:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:39:25.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:39:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72368128.0,"last_snapshot_bytes":72368128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"} 2024-09-16T12:39:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:39:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:39:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:39:25.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:39:25.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:39:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:39:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:39:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:39:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:39:23' 2024-09-16T12:39:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-16 12:39:23' 2024-09-16T12:39:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:39:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:39:23' 2024-09-16T12:39:25.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:39:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:39:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:39:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK942544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:23image-primary2513096d2-ff05-4526-8ac4-746ec0e8d36cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72368128.0,"last_snapshot_bytes":72368128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:23image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72364032.0,"last_snapshot_bytes":72364032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490265,"remote_snapshot_timestamp":1726490265,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:23image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64479232.0,"last_snapshot_bytes":64479232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:23image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70144000.0,"last_snapshot_bytes":70144000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:23image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63856640.0,"last_snapshot_bytes":63856640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490265,"remote_snapshot_timestamp":1726490265,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:23image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72273920.0,"last_snapshot_bytes":72273920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:23image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70205440.0,"last_snapshot_bytes":70205440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:23image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70234112.0,"last_snapshot_bytes":70234112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:23' 2024-09-16T12:39:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:39:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T12:39:25.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:39:23' 2024-09-16T12:39:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T12:39:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:39:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:39:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:39:25.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:39:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-16T12:39:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:39:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:25.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:39:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-16T12:39:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-16T12:39:25.934 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:39:25.923+0000 7f4071c2f700 -1 reset not still connected to 0x563591d7d040 2024-09-16T12:39:25.935 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:39:25.923+0000 7f4071c2f700 -1 reset not still connected to 0x56359eb99450 2024-09-16T12:39:26.036 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:39:26.023+0000 7ff4f6946700 -1 reset not still connected to 0x557568c81d40 2024-09-16T12:39:27.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:39:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-16T12:39:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:39:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:39:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:39:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary2 2024-09-16T12:39:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:27.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary2 2024-09-16T12:39:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:39:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary3 2024-09-16T12:39:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-16T12:39:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:39:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:39:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:39:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:39:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:39:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-16T12:39:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:39:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:39:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:39:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:39:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:39:28.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T12:39:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T12:39:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 2d9ab414-6b90-42a5-a42c-00a9400d330b 2024-09-16T12:39:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:39:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72364032.0,"last_snapshot_bytes":72364032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490265,"remote_snapshot_timestamp":1726490265,"replay_state":"idle"} 2024-09-16T12:39:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:39:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:39:23 2024-09-16T12:39:28.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:39:28.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:39:28.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:39:28.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:39:28.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:39:23' 2024-09-16T12:39:28.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:39:28.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:39:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:39:28.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72364032.0,"last_snapshot_bytes":72364032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490265,"remote_snapshot_timestamp":1726490265,"replay_state":"idle"} 2024-09-16T12:39:28.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:39:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:39:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:39:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:39:28.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:39:28.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:39:28.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:39:28.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:39:28.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:39:23' 2024-09-16T12:39:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-16 12:39:23' 2024-09-16T12:39:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:39:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:39:23' 2024-09-16T12:39:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:39:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:39:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:39:28.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK842544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:23image-primary32d9ab414-6b90-42a5-a42c-00a9400d330bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72364032.0,"last_snapshot_bytes":72364032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490265,"remote_snapshot_timestamp":1726490265,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:23image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64479232.0,"last_snapshot_bytes":64479232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:23image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70144000.0,"last_snapshot_bytes":70144000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:23image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63856640.0,"last_snapshot_bytes":63856640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490265,"remote_snapshot_timestamp":1726490265,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:23image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72273920.0,"last_snapshot_bytes":72273920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:23image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70205440.0,"last_snapshot_bytes":70205440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:23image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70234112.0,"last_snapshot_bytes":70234112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:23' 2024-09-16T12:39:28.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:39:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T12:39:28.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:39:23' 2024-09-16T12:39:28.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T12:39:28.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:39:28.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:39:28.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:39:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:39:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-16T12:39:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:39:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:28.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:39:28.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-16T12:39:28.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-16T12:39:29.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:39:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-16T12:39:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:39:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:39:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:39:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary3 2024-09-16T12:39:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary3 2024-09-16T12:39:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:39:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary4 2024-09-16T12:39:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-16T12:39:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:39:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:39:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:39:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:39:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:39:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-16T12:39:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:39:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:39:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:39:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:39:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:39:30.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T12:39:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T12:39:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 7c22ff11-90eb-4a64-9753-ee0488c7c6f0 2024-09-16T12:39:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:39:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64479232.0,"last_snapshot_bytes":64479232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"} 2024-09-16T12:39:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:39:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:39:23 2024-09-16T12:39:30.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:39:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:39:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:39:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:39:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:39:28' 2024-09-16T12:39:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:39:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:39:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:39:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64479232.0,"last_snapshot_bytes":64479232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"} 2024-09-16T12:39:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:39:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:39:30.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:39:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:39:30.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:39:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:39:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:39:30.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:39:30.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:39:23' 2024-09-16T12:39:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-16 12:39:23' 2024-09-16T12:39:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:39:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:39:23' 2024-09-16T12:39:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:39:30.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:39:30.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:39:30.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK742544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:28image-primary47c22ff11-90eb-4a64-9753-ee0488c7c6f0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":64479232.0,"last_snapshot_bytes":64479232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:28image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70144000.0,"last_snapshot_bytes":70144000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:28image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63856640.0,"last_snapshot_bytes":63856640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490265,"remote_snapshot_timestamp":1726490265,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:28image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72273920.0,"last_snapshot_bytes":72273920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:28image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70205440.0,"last_snapshot_bytes":70205440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:28image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70234112.0,"last_snapshot_bytes":70234112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:28' 2024-09-16T12:39:30.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:39:30.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T12:39:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:39:23' 2024-09-16T12:39:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T12:39:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:39:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:39:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:39:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:39:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-16T12:39:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:39:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:39:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-16T12:39:30.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-16T12:39:32.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:39:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-16T12:39:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:39:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:39:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:39:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary4 2024-09-16T12:39:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:32.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary4 2024-09-16T12:39:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:39:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary5 2024-09-16T12:39:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-16T12:39:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:39:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:39:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:39:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:39:32.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:39:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-16T12:39:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:33.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:39:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:39:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:39:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:39:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:39:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T12:39:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T12:39:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b2fd01b8-9e0f-4858-992e-38984ef0b6a3 2024-09-16T12:39:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:39:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70144000.0,"last_snapshot_bytes":70144000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"} 2024-09-16T12:39:33.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:39:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:39:23 2024-09-16T12:39:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:39:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:39:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:39:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:39:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:39:28' 2024-09-16T12:39:33.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:39:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:39:33.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:39:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70144000.0,"last_snapshot_bytes":70144000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"} 2024-09-16T12:39:33.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:39:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:39:33.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:39:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:39:33.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:39:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:39:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:39:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:39:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:39:23' 2024-09-16T12:39:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-16 12:39:23' 2024-09-16T12:39:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:39:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:39:23' 2024-09-16T12:39:33.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:39:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:39:33.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:39:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK642544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:28image-primary5b2fd01b8-9e0f-4858-992e-38984ef0b6a3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70144000.0,"last_snapshot_bytes":70144000,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:28image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63856640.0,"last_snapshot_bytes":63856640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490265,"remote_snapshot_timestamp":1726490265,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:28image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72273920.0,"last_snapshot_bytes":72273920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:28image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70205440.0,"last_snapshot_bytes":70205440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:28image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70234112.0,"last_snapshot_bytes":70234112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:28' 2024-09-16T12:39:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:39:33.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T12:39:33.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:39:23' 2024-09-16T12:39:33.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T12:39:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:39:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:39:33.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:39:33.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:39:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-16T12:39:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:39:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:39:33.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-16T12:39:33.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-16T12:39:33.979 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:39:33.967+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174d00 2024-09-16T12:39:33.979 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:39:33.967+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T12:39:35.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:39:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-16T12:39:35.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:39:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:39:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary5 2024-09-16T12:39:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:39:35.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:35.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary5 2024-09-16T12:39:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:39:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary6 2024-09-16T12:39:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-16T12:39:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:39:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:39:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:39:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:39:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:39:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-16T12:39:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:39:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:39:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:39:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:39:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:39:36.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T12:39:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T12:39:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 1166f0a7-946b-4553-bdb8-54616925dac3 2024-09-16T12:39:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:39:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63856640.0,"last_snapshot_bytes":63856640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490265,"remote_snapshot_timestamp":1726490265,"replay_state":"idle"} 2024-09-16T12:39:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:39:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:39:23 2024-09-16T12:39:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:39:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:39:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:39:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:39:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:39:33' 2024-09-16T12:39:36.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:39:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:39:36.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:39:36.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63856640.0,"last_snapshot_bytes":63856640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490265,"remote_snapshot_timestamp":1726490265,"replay_state":"idle"} 2024-09-16T12:39:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:39:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:39:36.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:39:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:39:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:39:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:39:36.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:39:36.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:39:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:39:23' 2024-09-16T12:39:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-16 12:39:23' 2024-09-16T12:39:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:39:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:39:23' 2024-09-16T12:39:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:39:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:39:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:39:36.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK542544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:33image-primary61166f0a7-946b-4553-bdb8-54616925dac3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63856640.0,"last_snapshot_bytes":63856640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490265,"remote_snapshot_timestamp":1726490265,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:33image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72273920.0,"last_snapshot_bytes":72273920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:33image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70205440.0,"last_snapshot_bytes":70205440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:33image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70234112.0,"last_snapshot_bytes":70234112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:33' 2024-09-16T12:39:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:39:36.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T12:39:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:39:23' 2024-09-16T12:39:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T12:39:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:39:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:39:36.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:39:36.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:39:36.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:36.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:36.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-16T12:39:36.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:39:36.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:39:36.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-16T12:39:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-16T12:39:38.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:39:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-16T12:39:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:39:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:39:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:39:38.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary6 2024-09-16T12:39:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary6 2024-09-16T12:39:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:39:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary7 2024-09-16T12:39:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-16T12:39:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:39:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:39:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:39:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:39:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:39:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-16T12:39:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:39:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:39:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:39:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:39:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:39:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T12:39:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T12:39:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 84b20afe-0f64-4172-b43d-9f6a37443624 2024-09-16T12:39:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:39:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72273920.0,"last_snapshot_bytes":72273920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"} 2024-09-16T12:39:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:39:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:39:23 2024-09-16T12:39:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:39:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:39:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:39:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:39:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:39:38' 2024-09-16T12:39:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:39:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:39:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:39:39.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72273920.0,"last_snapshot_bytes":72273920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"} 2024-09-16T12:39:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:39:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:39:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:39:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:39:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:39:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:39:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:39:39.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:39:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:39:23' 2024-09-16T12:39:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-16 12:39:23' 2024-09-16T12:39:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:39.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:39:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:39:23' 2024-09-16T12:39:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:39:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:39:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:39:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK442544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:38image-primary784b20afe-0f64-4172-b43d-9f6a37443624up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72273920.0,"last_snapshot_bytes":72273920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:38image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70205440.0,"last_snapshot_bytes":70205440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:38image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70234112.0,"last_snapshot_bytes":70234112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:38' 2024-09-16T12:39:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:39:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T12:39:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:39:23' 2024-09-16T12:39:39.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T12:39:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:39:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:39:39.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:39:39.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:39:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-16T12:39:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:39:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:39:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-16T12:39:39.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-16T12:39:40.785 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:39:40.771+0000 7f61b1855700 -1 reset not still connected to 0x557b229664e0 2024-09-16T12:39:40.785 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:39:40.771+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T12:39:40.785 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:39:40.771+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T12:39:40.785 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:39:40.771+0000 7f61b1855700 -1 reset not still connected to 0x557b3da4b450 2024-09-16T12:39:40.785 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:39:40.771+0000 7f61b1855700 -1 reset not still connected to 0x557b40431c70 2024-09-16T12:39:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:39:41.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-16T12:39:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:39:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:39:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:39:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary7 2024-09-16T12:39:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary7 2024-09-16T12:39:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:39:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary8 2024-09-16T12:39:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-16T12:39:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:39:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:39:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:39:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:39:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:39:42.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-16T12:39:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:39:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:39:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:39:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:39:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:39:42.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T12:39:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T12:39:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 03a4f57d-6ac5-4d45-8468-694f291bfc5a 2024-09-16T12:39:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:39:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70205440.0,"last_snapshot_bytes":70205440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"} 2024-09-16T12:39:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:39:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:39:23 2024-09-16T12:39:42.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:39:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:39:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:39:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:39:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:39:38' 2024-09-16T12:39:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:39:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:39:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:39:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70205440.0,"last_snapshot_bytes":70205440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"} 2024-09-16T12:39:42.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:39:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:39:42.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:39:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:39:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:39:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:39:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:39:42.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:39:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:39:23' 2024-09-16T12:39:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-16 12:39:23' 2024-09-16T12:39:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:39:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:39:23' 2024-09-16T12:39:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:39:42.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:39:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:39:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK342544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:38image-primary803a4f57d-6ac5-4d45-8468-694f291bfc5aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70205440.0,"last_snapshot_bytes":70205440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:38image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70234112.0,"last_snapshot_bytes":70234112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:38' 2024-09-16T12:39:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:39:42.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T12:39:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:39:23' 2024-09-16T12:39:42.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T12:39:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:39:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:39:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:39:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:39:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-16T12:39:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:39:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:39:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-16T12:39:42.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-16T12:39:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:39:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-16T12:39:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:39:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:39:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:39:44.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary8 2024-09-16T12:39:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:44.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary8 2024-09-16T12:39:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:39:44.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary9 2024-09-16T12:39:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-16T12:39:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:39:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:39:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:39:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:39:44.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:39:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-16T12:39:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:39:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:45.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:39:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:39:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:39:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:39:45.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T12:39:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T12:39:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 6588a7f2-9bf0-4c8d-9a72-4664809bda6b 2024-09-16T12:39:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:39:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70234112.0,"last_snapshot_bytes":70234112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"} 2024-09-16T12:39:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:39:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:39:23 2024-09-16T12:39:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:39:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:39:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:39:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:39:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:39:43' 2024-09-16T12:39:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:39:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:39:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:39:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70234112.0,"last_snapshot_bytes":70234112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"} 2024-09-16T12:39:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:39:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:39:45.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:39:45.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:39:45.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:39:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:39:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:39:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:39:23' 2024-09-16T12:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-16 12:39:23' 2024-09-16T12:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:39:23' 2024-09-16T12:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:39:45.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:39:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK242544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:43image-primary96588a7f2-9bf0-4c8d-9a72-4664809bda6bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70234112.0,"last_snapshot_bytes":70234112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:43' 2024-09-16T12:39:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:39:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T12:39:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:39:23' 2024-09-16T12:39:45.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T12:39:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:39:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:39:45.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:39:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:39:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-16T12:39:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:39:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:39:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-16T12:39:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-16T12:39:47.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:39:47.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-16T12:39:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:39:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:39:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:39:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary9 2024-09-16T12:39:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:47.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary9 2024-09-16T12:39:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:39:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary10 2024-09-16T12:39:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-16T12:39:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:39:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:39:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:39:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:39:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:39:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-16T12:39:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:39:48.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:39:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:39:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:39:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:39:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T12:39:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T12:39:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: f67b4d08-9215-4c3a-be5a-c06ea7bbbb73 2024-09-16T12:39:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:39:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"} 2024-09-16T12:39:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:39:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:39:23 2024-09-16T12:39:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:39:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:39:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:39:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:39:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:39:48' 2024-09-16T12:39:48.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:39:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:39:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:39:48.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"} 2024-09-16T12:39:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:39:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:39:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:39:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:39:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:39:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:39:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:39:48.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:39:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:39:23' 2024-09-16T12:39:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-16 12:39:23' 2024-09-16T12:39:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:39:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:39:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:39:23' 2024-09-16T12:39:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:39:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:39:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:39:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK142544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10f67b4d08-9215-4c3a-be5a-c06ea7bbbb73up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490262,"remote_snapshot_timestamp":1726490262,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:39:48' 2024-09-16T12:39:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:39:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T12:39:49.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:39:23' 2024-09-16T12:39:49.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T12:39:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:39:49.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:39:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:39:49.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:39:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-16T12:39:49.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:39:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:39:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-16T12:39:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-16T12:39:51.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:39:51.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-16T12:39:51.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:51.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:51.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:39:51.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:39:51.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary10 2024-09-16T12:39:51.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:39:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:39:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary10 2024-09-16T12:39:51.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..10} 2024-09-16T12:39:51.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS=() 2024-09-16T12:39:51.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS=() 2024-09-16T12:39:51.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS=() 2024-09-16T12:39:51.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS=() 2024-09-16T12:39:51.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ RET=0 2024-09-16T12:39:51.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:39:51.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary1 2024-09-16T12:39:51.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary1 2024-09-16T12:39:51.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-16T12:39:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:39:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:39:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:39:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-16T12:39:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:39:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:39:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:39:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:39:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-16T12:39:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:39:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-16T12:39:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:39:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:39:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-16T12:39:51.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:39:51.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-16T12:39:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary1': 2024-09-16T12:39:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:39:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:39:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:39:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 19f7100bacf5 2024-09-16T12:39:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.19f7100bacf5 2024-09-16T12:39:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:39:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:39:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:39:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:39:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:39:51 2024 2024-09-16T12:39:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:39:51 2024 2024-09-16T12:39:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:39:51 2024 2024-09-16T12:39:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:39:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:39:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 00da4166-a0f4-4e0d-a0cb-e3ce5afbffa1 2024-09-16T12:39:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:39:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:39:51.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary1 2024-09-16T12:39:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd0 2024-09-16T12:39:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:39:52.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd0 2024-09-16T12:39:52.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:39:52.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:39:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:39:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: c4ff484b-fa02-49ef-a469-1b9c598ae533 2024-09-16T12:39:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:39:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:39:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:39:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:39:52.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:39:52.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:39:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:39:52.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:39:52.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary1 2024-09-16T12:39:52.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd0 test-primary1 2024-09-16T12:39:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:39:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary1 2024-09-16T12:39:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:39:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:39:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:39:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-16T12:39:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary2 2024-09-16T12:39:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary1 2024-09-16T12:39:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary2 2024-09-16T12:39:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-16T12:39:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary1 2024-09-16T12:39:52.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-16T12:39:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:39:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:39:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:39:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:39:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:39:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:39:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:52.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:39:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:39:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-16T12:39:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:39:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:39:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:39:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:39:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:52.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-16T12:39:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:39:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-16T12:39:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:39:52.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:39:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-16T12:39:52.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:39:52.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-16T12:39:53.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1319 2024-09-16T12:39:53.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:39:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:39:53.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-16T12:39:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary2': 2024-09-16T12:39:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:39:53.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:39:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:39:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1a06a15649c8 2024-09-16T12:39:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1a06a15649c8 2024-09-16T12:39:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:39:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:39:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:39:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:39:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:39:52 2024 2024-09-16T12:39:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:39:52 2024 2024-09-16T12:39:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:39:52 2024 2024-09-16T12:39:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:39:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:39:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: fa948dbe-a7fa-43eb-b157-1a11a03be35d 2024-09-16T12:39:53.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:39:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:39:53.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary2 2024-09-16T12:39:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd1 2024-09-16T12:39:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:39:54.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd1 2024-09-16T12:39:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:39:54.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:39:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:39:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 7bd449d4-c4a8-41f9-a4b5-92066ceea74b 2024-09-16T12:39:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:39:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:39:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:39:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:39:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:39:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:39:54.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:39:54.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:39:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary2 2024-09-16T12:39:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd1 test-primary2 2024-09-16T12:39:54.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:39:54.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary2 2024-09-16T12:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-16T12:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary2 2024-09-16T12:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:39:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary3 2024-09-16T12:39:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary3 2024-09-16T12:39:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:39:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-16T12:39:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary2 2024-09-16T12:39:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:39:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-16T12:39:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:39:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:39:54.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:39:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-16T12:39:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:39:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:39:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:39:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:39:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-16T12:39:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:39:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-16T12:39:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:54.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:39:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:39:54.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-16T12:39:54.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:39:54.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-16T12:39:54.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1323 2024-09-16T12:39:54.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:39:54.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:39:54.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-16T12:39:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary3': 2024-09-16T12:39:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:39:54.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:39:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:39:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1a16e0b629a4 2024-09-16T12:39:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1a16e0b629a4 2024-09-16T12:39:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:39:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:39:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:39:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:39:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:39:54 2024 2024-09-16T12:39:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:39:54 2024 2024-09-16T12:39:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:39:54 2024 2024-09-16T12:39:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:39:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:39:54.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 04708389-b988-49ba-9b70-98157102cb71 2024-09-16T12:39:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:39:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:39:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary3 2024-09-16T12:39:55.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd2 2024-09-16T12:39:55.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:39:55.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd2 2024-09-16T12:39:55.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:39:55.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:39:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:39:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: ea14f0c3-c465-4b24-adba-3a3003c31b3f 2024-09-16T12:39:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:39:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:39:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:39:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:39:55.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:39:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:39:55.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:39:55.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:39:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary3 2024-09-16T12:39:55.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd2 test-primary3 2024-09-16T12:39:55.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:39:55.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary3 2024-09-16T12:39:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:39:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-16T12:39:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:39:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:39:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:39:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:39:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:39:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:39:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary3 2024-09-16T12:39:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:39:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary4 2024-09-16T12:39:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary4 2024-09-16T12:39:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary3 2024-09-16T12:39:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-16T12:39:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-16T12:39:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:55.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:39:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:39:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:39:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-16T12:39:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:39:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:39:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:39:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:39:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-16T12:39:55.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:39:55.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-16T12:39:55.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:55.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:55.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:39:55.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:39:55.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-16T12:39:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:39:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-16T12:39:55.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1326 2024-09-16T12:39:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:39:55.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:39:55.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-16T12:39:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary4': 2024-09-16T12:39:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:39:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:39:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:39:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1a26af0e7307 2024-09-16T12:39:55.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1a26af0e7307 2024-09-16T12:39:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:39:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:39:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:39:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:39:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:39:55 2024 2024-09-16T12:39:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:39:55 2024 2024-09-16T12:39:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:39:55 2024 2024-09-16T12:39:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:39:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:39:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 9e488d88-4f84-437a-a54a-0803bfdd4803 2024-09-16T12:39:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:39:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:39:55.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary4 2024-09-16T12:39:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd3 2024-09-16T12:39:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:39:56.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd3 2024-09-16T12:39:56.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:39:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:39:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:39:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: b5ad7b88-5728-47bc-9ee7-b45cf1b88f13 2024-09-16T12:39:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:39:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:39:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:39:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:39:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:39:56.335 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:39:56.323+0000 7f4071c2f700 -1 reset not still connected to 0x563591d7d040 2024-09-16T12:39:56.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:39:56.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:39:56.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:39:56.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary4 2024-09-16T12:39:56.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd3 test-primary4 2024-09-16T12:39:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:39:56.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary4 2024-09-16T12:39:56.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:39:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-16T12:39:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:39:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:39:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:39:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary5 2024-09-16T12:39:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary5 2024-09-16T12:39:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-16T12:39:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:39:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:39:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:56.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:39:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:39:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:39:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:39:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:39:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-16T12:39:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary4 2024-09-16T12:39:56.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:39:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:39:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:39:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:39:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-16T12:39:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary4 2024-09-16T12:39:56.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-16T12:39:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:39:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-16T12:39:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:39:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:39:56.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-16T12:39:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:39:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-16T12:39:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:39:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:39:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:39:56.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:39:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1328 2024-09-16T12:39:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:39:57.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:39:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:39:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:39:57.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:57.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:57.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:39:57.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:39:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-16T12:39:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary5': 2024-09-16T12:39:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:39:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:39:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:39:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1a342069f7e5 2024-09-16T12:39:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1a342069f7e5 2024-09-16T12:39:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:39:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:39:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:39:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:39:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:39:56 2024 2024-09-16T12:39:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:39:56 2024 2024-09-16T12:39:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:39:56 2024 2024-09-16T12:39:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:39:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:39:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 18ee13c8-2f19-45e6-885e-456ffd58cc67 2024-09-16T12:39:57.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:39:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:39:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary5 2024-09-16T12:39:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd4 2024-09-16T12:39:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:39:58.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd4 2024-09-16T12:39:58.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:39:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:39:58.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:39:58.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 27eec49a-8e7c-4bb9-9c90-63e80d20a8e4 2024-09-16T12:39:58.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:39:58.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:39:58.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:39:58.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:39:58.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:39:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:39:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:39:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:39:58.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary5 2024-09-16T12:39:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd4 test-primary5 2024-09-16T12:39:58.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:39:58.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary5 2024-09-16T12:39:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:39:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-16T12:39:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:39:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:39:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:39:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:39:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:39:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary5 2024-09-16T12:39:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary6 2024-09-16T12:39:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary6 2024-09-16T12:39:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-16T12:39:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary5 2024-09-16T12:39:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-16T12:39:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:39:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:39:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:39:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-16T12:39:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:39:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:39:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:39:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:39:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-16T12:39:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:39:58.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:39:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:39:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-16T12:39:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:39:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:39:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-16T12:39:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:39:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-16T12:39:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1330 2024-09-16T12:39:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:39:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:39:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:39:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:39:58.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:39:58.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1335 2024-09-16T12:39:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:39:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:39:58.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-16T12:39:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary6': 2024-09-16T12:39:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:39:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:39:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:39:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1a49c702c486 2024-09-16T12:39:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1a49c702c486 2024-09-16T12:39:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:39:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:39:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:39:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:39:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:39:58 2024 2024-09-16T12:39:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:39:58 2024 2024-09-16T12:39:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:39:58 2024 2024-09-16T12:39:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:39:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:39:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 18c1291b-9eb8-46a9-82b4-f0d012b14821 2024-09-16T12:39:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:39:58.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:39:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary6 2024-09-16T12:39:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd5 2024-09-16T12:39:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:39:59.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd5 2024-09-16T12:39:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:39:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:39:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:39:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: f7881da7-198c-4b7c-91ab-b471199a2166 2024-09-16T12:39:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:39:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:39:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:39:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:39:59.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:39:59.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:39:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:39:59.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:39:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary6 2024-09-16T12:39:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd5 test-primary6 2024-09-16T12:39:59.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:39:59.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary6 2024-09-16T12:39:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:39:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-16T12:39:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:39:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:39:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:39:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:39:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:39:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary6 2024-09-16T12:39:59.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary7 2024-09-16T12:39:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary7 2024-09-16T12:39:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:39:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-16T12:39:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:39:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary6 2024-09-16T12:39:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-16T12:39:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:39:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:39:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:39:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-16T12:39:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:39:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:39:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:39:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:39:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:39:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-16T12:39:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:39:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-16T12:39:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:39:59.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:39:59.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-16T12:39:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:39:59.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-16T12:39:59.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1334 2024-09-16T12:39:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:39:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:39:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:39:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:39:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:39:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:39:59.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:39:59.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1340 2024-09-16T12:39:59.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:39:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:39:59.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-16T12:40:00.057 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:40:00.043+0000 7f61b1855700 -1 reset not still connected to 0x557b229664e0 2024-09-16T12:40:00.058 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:40:00.043+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T12:40:00.058 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:40:00.047+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T12:40:00.058 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:40:00.047+0000 7f61b1855700 -1 reset not still connected to 0x557b28f83c70 2024-09-16T12:40:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary7': 2024-09-16T12:40:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:40:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:40:00.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:40:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1a5b23b89725 2024-09-16T12:40:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1a5b23b89725 2024-09-16T12:40:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:40:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:40:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:40:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:40:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:39:59 2024 2024-09-16T12:40:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:39:59 2024 2024-09-16T12:40:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:39:59 2024 2024-09-16T12:40:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:40:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:40:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: a80f552e-4c2a-48a2-8fa0-3670fc11d7e8 2024-09-16T12:40:00.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:40:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:40:00.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary7 2024-09-16T12:40:00.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd6 2024-09-16T12:40:00.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:40:00.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd6 2024-09-16T12:40:00.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:40:00.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:40:00.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:40:00.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 1140b5b9-21cc-4dfd-9b65-cfb057a6b382 2024-09-16T12:40:00.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:40:00.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:40:00.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:40:00.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:40:00.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:40:00.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:40:00.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:40:00.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:40:00.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary7 2024-09-16T12:40:00.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd6 test-primary7 2024-09-16T12:40:00.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1339 2024-09-16T12:40:00.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:40:00.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary7 2024-09-16T12:40:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:40:00.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:40:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:40:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary8 2024-09-16T12:40:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary8 2024-09-16T12:40:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-16T12:40:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary7 2024-09-16T12:40:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:00.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:40:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:00.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:40:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:40:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-16T12:40:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:00.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:40:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:40:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:40:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:40:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-16T12:40:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary7 2024-09-16T12:40:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-16T12:40:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-16T12:40:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:00.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:40:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:40:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:40:00.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:40:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:40:01.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-16T12:40:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:40:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-16T12:40:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:40:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:40:01.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-16T12:40:01.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1342 2024-09-16T12:40:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:40:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:40:01.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:40:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:40:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:40:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:40:01.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:40:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1345 2024-09-16T12:40:01.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-16T12:40:01.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:01.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary8': 2024-09-16T12:40:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:40:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:40:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:40:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1a6e6ec2ddfd 2024-09-16T12:40:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1a6e6ec2ddfd 2024-09-16T12:40:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:40:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:40:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:40:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:40:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:40:01 2024 2024-09-16T12:40:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:40:01 2024 2024-09-16T12:40:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:40:01 2024 2024-09-16T12:40:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:40:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:40:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 387ce0f1-7b85-4979-8cef-4635c6f65150 2024-09-16T12:40:01.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:40:01.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:40:01.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary8 2024-09-16T12:40:02.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd7 2024-09-16T12:40:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:40:02.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd7 2024-09-16T12:40:02.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:40:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:40:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:40:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 8a161be0-25f6-4c0e-bcac-18925d52c95b 2024-09-16T12:40:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:40:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:40:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:40:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:40:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:40:02.380 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:40:02.367+0000 7f61b1855700 -1 reset not still connected to 0x557b229665b0 2024-09-16T12:40:02.380 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:40:02.367+0000 7f61b1855700 -1 reset not still connected to 0x557b22af11e0 2024-09-16T12:40:02.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:40:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:40:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:40:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary8 2024-09-16T12:40:02.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd7 test-primary8 2024-09-16T12:40:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:40:02.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary8 2024-09-16T12:40:02.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:40:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-16T12:40:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:40:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:40:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:40:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:40:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary8 2024-09-16T12:40:02.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:40:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:40:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary9 2024-09-16T12:40:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary9 2024-09-16T12:40:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-16T12:40:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary8 2024-09-16T12:40:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-16T12:40:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:40:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:02.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:40:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:40:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-16T12:40:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:40:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:40:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:40:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:40:02.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:02.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-16T12:40:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:40:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-16T12:40:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:40:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:40:02.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-16T12:40:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:40:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-16T12:40:02.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1349 2024-09-16T12:40:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:40:02.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:02.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-16T12:40:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:40:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:40:02.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:40:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary9': 2024-09-16T12:40:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:40:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:40:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:40:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1a833c8bde09 2024-09-16T12:40:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1a833c8bde09 2024-09-16T12:40:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:40:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:40:02.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:40:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:40:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:40:02 2024 2024-09-16T12:40:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:40:02 2024 2024-09-16T12:40:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:40:02 2024 2024-09-16T12:40:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:40:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:40:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: f057b092-1b46-4f3e-b5ba-d6581e9d4b6c 2024-09-16T12:40:02.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:40:02.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:40:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary9 2024-09-16T12:40:02.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:40:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:40:02.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:40:03.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd8 2024-09-16T12:40:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:40:03.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd8 2024-09-16T12:40:03.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:40:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:40:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:40:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: dd57f505-422e-49ce-93db-fcacaea222d7 2024-09-16T12:40:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:40:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:40:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:40:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:40:03.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:40:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:40:03.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:40:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:40:03.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary9 2024-09-16T12:40:03.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd8 test-primary9 2024-09-16T12:40:03.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:40:03.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary9 2024-09-16T12:40:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:40:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-16T12:40:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:40:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:40:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:40:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:40:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:40:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:40:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary10 2024-09-16T12:40:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary10 2024-09-16T12:40:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-16T12:40:03.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:40:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:40:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:40:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-16T12:40:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary9 2024-09-16T12:40:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:40:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:40:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:40:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:40:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:40:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-16T12:40:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary9 2024-09-16T12:40:03.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-16T12:40:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:40:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-16T12:40:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:40:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:40:03.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-16T12:40:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:40:03.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-16T12:40:03.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1348 2024-09-16T12:40:03.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:40:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:40:03.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:40:03.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1347 2024-09-16T12:40:03.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1355 2024-09-16T12:40:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:40:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:40:04.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:40:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1354 2024-09-16T12:40:04.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:04.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1353 2024-09-16T12:40:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:40:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:04.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:40:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:40:04.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:04.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:40:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-16T12:40:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary10': 2024-09-16T12:40:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:40:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:40:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:40:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1a9663001776 2024-09-16T12:40:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1a9663001776 2024-09-16T12:40:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:40:05.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:40:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:40:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:40:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:40:03 2024 2024-09-16T12:40:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:40:03 2024 2024-09-16T12:40:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:40:03 2024 2024-09-16T12:40:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:40:05.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:40:05.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: cfa45128-953f-406f-a7c1-a2354ab18676 2024-09-16T12:40:05.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:40:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:40:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary10 2024-09-16T12:40:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd9 2024-09-16T12:40:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:40:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd9 2024-09-16T12:40:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:40:05.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:40:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:40:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 73c69c2e-2c3c-4ccc-9d8b-b9afd21d9102 2024-09-16T12:40:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:40:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:40:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:40:05.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:40:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:40:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:40:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:40:05.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:40:05.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary10 2024-09-16T12:40:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd9 test-primary10 2024-09-16T12:40:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:40:05.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:40:05.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:40:05.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1359 2024-09-16T12:40:05.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:40:05.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary10 2024-09-16T12:40:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:40:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-16T12:40:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:40:05.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:40:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:40:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 51084 2024-09-16T12:40:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:40:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:40:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:40:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary10 2024-09-16T12:40:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary10 2024-09-16T12:40:05.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-16T12:40:06.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:40:06.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-16T12:40:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:40:06.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:40:06.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:40:06.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1360 2024-09-16T12:40:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:40:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:06.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:40:06.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:40:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:40:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:40:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1362 2024-09-16T12:40:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:40:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1365 2024-09-16T12:40:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:07.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1363 2024-09-16T12:40:07.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:40:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:40:07.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:40:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:40:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:40:07.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:40:08.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1367 2024-09-16T12:40:08.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:40:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:40:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:40:08.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1369 2024-09-16T12:40:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1368 2024-09-16T12:40:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:08.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:40:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:09.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:40:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:40:09.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1375 2024-09-16T12:40:09.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1374 2024-09-16T12:40:09.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:09.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:40:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:40:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:40:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:40:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:40:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:40:10.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:40:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:40:10.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:40:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1378 2024-09-16T12:40:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:40:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:40:11.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:40:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1381 2024-09-16T12:40:11.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1380 2024-09-16T12:40:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:11.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1382 2024-09-16T12:40:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:40:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:40:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:40:11.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:40:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:40:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:40:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1385 2024-09-16T12:40:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:40:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:40:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:40:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:40:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:40:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:40:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:40:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:40:13.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:40:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1390 2024-09-16T12:40:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1388 2024-09-16T12:40:13.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:13.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1389 2024-09-16T12:40:13.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1394 2024-09-16T12:40:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1395 2024-09-16T12:40:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:40:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:40:14.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:40:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:14.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:14.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:14.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:40:14.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:14.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:14.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:40:14.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:40:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:40:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:40:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:40:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:40:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:40:15.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:40:15.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1396 2024-09-16T12:40:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1400 2024-09-16T12:40:16.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1402 2024-09-16T12:40:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:40:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:40:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:40:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1401 2024-09-16T12:40:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:40:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:40:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:40:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:40:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:40:16.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:40:17.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:40:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:40:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:40:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:40:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:40:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:40:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1403 2024-09-16T12:40:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:40:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:40:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:40:18.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1410 2024-09-16T12:40:18.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1408 2024-09-16T12:40:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1409 2024-09-16T12:40:18.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:18.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:18.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:19.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:40:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:40:19.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:40:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:40:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:40:19.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:40:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1415 2024-09-16T12:40:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:40:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:40:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:40:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1414 2024-09-16T12:40:19.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:19.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1416 2024-09-16T12:40:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:40:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:40:20.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:40:20.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1419 2024-09-16T12:40:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:40:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:40:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:40:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:40:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:40:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:40:21.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:40:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:40:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:40:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1421 2024-09-16T12:40:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1422 2024-09-16T12:40:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1426 2024-09-16T12:40:22.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:40:22.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:40:22.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:40:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:40:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:40:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:40:23.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1429 2024-09-16T12:40:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1428 2024-09-16T12:40:23.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1430 2024-09-16T12:40:23.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:23.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:23.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:40:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:40:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:40:24.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1434 2024-09-16T12:40:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1435 2024-09-16T12:40:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:24.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:40:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:40:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:40:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:40:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:40:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:40:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:40:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:40:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:40:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:40:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:40:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:40:25.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1438 2024-09-16T12:40:25.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1442 2024-09-16T12:40:26.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1441 2024-09-16T12:40:26.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:26.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:40:26.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:26.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:26.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:40:26.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:40:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1440 2024-09-16T12:40:26.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:40:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:40:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:40:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:40:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:26.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:40:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:40:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:26.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1446 2024-09-16T12:40:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:40:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:40:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:40:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:40:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:40:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:40:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:40:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:40:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:40:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1450 2024-09-16T12:40:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1449 2024-09-16T12:40:28.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1447 2024-09-16T12:40:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:28.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:28.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:29.390 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T12:40:29.378+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8d67cc9c0 2024-09-16T12:40:29.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:40:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:40:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:40:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:40:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:40:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:40:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1453 2024-09-16T12:40:29.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1454 2024-09-16T12:40:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:40:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:40:29.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:40:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:30.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:40:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:40:30.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:40:30.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1457 2024-09-16T12:40:30.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:31.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1461 2024-09-16T12:40:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1462 2024-09-16T12:40:31.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1460 2024-09-16T12:40:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:40:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:40:31.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:40:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:40:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:31.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:40:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:40:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:31.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:40:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:40:31.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:40:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1464 2024-09-16T12:40:32.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:40:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:32.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:40:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:40:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:40:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:40:32.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:40:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:40:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:40:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:40:33.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1468 2024-09-16T12:40:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1470 2024-09-16T12:40:33.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:33.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1469 2024-09-16T12:40:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:33.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:34.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1475 2024-09-16T12:40:34.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:40:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:40:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:40:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:40:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:40:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:40:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:34.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:40:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:40:34.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:40:34.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1474 2024-09-16T12:40:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:40:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:40:35.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:40:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1478 2024-09-16T12:40:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:36.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:40:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:40:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:40:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1482 2024-09-16T12:40:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:40:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:40:36.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:40:36.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1480 2024-09-16T12:40:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:40:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:40:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:40:36.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1481 2024-09-16T12:40:36.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:36.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1483 2024-09-16T12:40:37.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:40:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:40:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:40:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:40:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:40:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:40:37.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:40:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:40:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:40:38.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1490 2024-09-16T12:40:38.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1487 2024-09-16T12:40:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:38.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1489 2024-09-16T12:40:38.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1494 2024-09-16T12:40:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:39.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:40:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:40:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:40:39.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1495 2024-09-16T12:40:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:40:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:40:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:40:39.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:40:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:40:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:40:39.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:40.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1496 2024-09-16T12:40:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:40.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:40:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:40:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:40:40.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:40:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:40:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:40:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:40:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:40:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:40:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1502 2024-09-16T12:40:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1501 2024-09-16T12:40:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:40:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:40:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:40:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:41.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1500 2024-09-16T12:40:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:42.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1505 2024-09-16T12:40:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:40:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:40:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:40:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:40:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:40:42.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:40:43.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:43.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:40:43.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:43.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:43.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:40:43.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:40:43.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1510 2024-09-16T12:40:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1509 2024-09-16T12:40:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:44.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1508 2024-09-16T12:40:44.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1515 2024-09-16T12:40:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1514 2024-09-16T12:40:44.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:40:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:40:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:40:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:40:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:40:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:40:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:40:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:40:44.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:40:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:40:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:40:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:40:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1518 2024-09-16T12:40:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1521 2024-09-16T12:40:46.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1522 2024-09-16T12:40:46.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1520 2024-09-16T12:40:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:40:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:40:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:40:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:46.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:40:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:40:46.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:40:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:40:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:40:47.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:40:47.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:47.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:40:47.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:47.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:47.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:40:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:40:47.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1523 2024-09-16T12:40:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:40:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:40:47.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:40:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:40:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:40:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:40:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1529 2024-09-16T12:40:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1530 2024-09-16T12:40:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1528 2024-09-16T12:40:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:40:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:49.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:40:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:40:49.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1535 2024-09-16T12:40:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:40:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:49.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:40:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:40:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:40:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:40:49.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:40:49.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:49.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1534 2024-09-16T12:40:49.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:50.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1536 2024-09-16T12:40:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:40:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:40:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:40:50.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1542 2024-09-16T12:40:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:40:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:40:51.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:40:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:51.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1541 2024-09-16T12:40:51.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1539 2024-09-16T12:40:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:40:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:40:51.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:40:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:52.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:40:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:40:52.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:40:52.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:52.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:52.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1546 2024-09-16T12:40:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:52.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:40:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:40:52.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:40:52.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:40:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:40:52.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:40:53.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1548 2024-09-16T12:40:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1550 2024-09-16T12:40:53.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:53.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:53.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1549 2024-09-16T12:40:53.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:40:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:53.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:40:53.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:40:54.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1554 2024-09-16T12:40:54.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1555 2024-09-16T12:40:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:54.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:40:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:40:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:40:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:40:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:40:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:40:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:40:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:40:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:40:55.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1558 2024-09-16T12:40:55.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:40:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:40:55.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:40:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:40:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:56.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:40:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:40:56.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1562 2024-09-16T12:40:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:40:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:40:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:40:56.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1561 2024-09-16T12:40:56.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:56.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:56.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1560 2024-09-16T12:40:56.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:40:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:40:56.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:40:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1566 2024-09-16T12:40:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:40:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:40:57.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:40:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:57.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:40:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:40:57.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:40:57.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:40:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:40:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:40:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1569 2024-09-16T12:40:58.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:58.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1568 2024-09-16T12:40:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:58.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1570 2024-09-16T12:40:58.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:40:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:40:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:40:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:40:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:40:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:40:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1574 2024-09-16T12:40:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1575 2024-09-16T12:40:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:40:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:40:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:40:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:40:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:40:59.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:40:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:40:59.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:41:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:41:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:41:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1576 2024-09-16T12:41:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:41:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:41:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:41:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1581 2024-09-16T12:41:01.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1582 2024-09-16T12:41:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:41:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:41:01.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:41:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1580 2024-09-16T12:41:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:41:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:41:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:41:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:02.466 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:41:02.455+0000 7f4071c2f700 -1 reset not still connected to 0x5635a8fa6270 2024-09-16T12:41:02.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1586 2024-09-16T12:41:02.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:02.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:02.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:41:02.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:02.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:02.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:41:02.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:41:02.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:02.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:41:02.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:02.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:02.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:41:02.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:41:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:41:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:03.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:41:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:41:03.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1587 2024-09-16T12:41:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1590 2024-09-16T12:41:03.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1588 2024-09-16T12:41:03.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:03.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:03.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1595 2024-09-16T12:41:04.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1594 2024-09-16T12:41:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:04.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:41:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:41:04.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:41:04.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:41:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:41:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:41:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:41:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:04.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:41:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:41:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:05.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:41:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:41:05.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:41:05.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1598 2024-09-16T12:41:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1601 2024-09-16T12:41:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1602 2024-09-16T12:41:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1600 2024-09-16T12:41:06.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:41:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:41:06.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:41:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:41:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:41:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:41:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:41:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:41:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:41:06.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:41:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:41:07.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:41:07.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1603 2024-09-16T12:41:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:41:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:07.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:41:07.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:41:07.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:08.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1608 2024-09-16T12:41:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:41:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:41:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:41:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:08.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1610 2024-09-16T12:41:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1609 2024-09-16T12:41:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:08.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:09.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1614 2024-09-16T12:41:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1615 2024-09-16T12:41:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:41:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:41:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:41:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:41:09.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:41:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:41:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:41:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:41:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:41:09.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:41:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:41:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:41:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1617 2024-09-16T12:41:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:41:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:41:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:41:11.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1620 2024-09-16T12:41:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:41:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:41:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:41:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1622 2024-09-16T12:41:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1621 2024-09-16T12:41:11.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:41:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:41:11.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:41:11.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1625 2024-09-16T12:41:12.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:12.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:41:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:41:12.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:41:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:41:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:41:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:41:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:41:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:41:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:41:13.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1628 2024-09-16T12:41:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1630 2024-09-16T12:41:13.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1629 2024-09-16T12:41:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:13.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:14.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1635 2024-09-16T12:41:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1634 2024-09-16T12:41:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:41:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:14.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:41:14.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:41:14.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:14.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:41:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:41:14.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:41:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:41:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:41:14.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:41:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:41:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:41:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:41:15.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1637 2024-09-16T12:41:15.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:15.988 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:41:15.975+0000 7f61b1855700 -1 reset not still connected to 0x557b39beb040 2024-09-16T12:41:16.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1642 2024-09-16T12:41:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1641 2024-09-16T12:41:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:41:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:41:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:41:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:41:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:41:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:41:16.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:41:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:41:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:41:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1640 2024-09-16T12:41:16.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:16.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:17.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:41:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:41:17.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:41:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:41:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:41:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:41:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1643 2024-09-16T12:41:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:41:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:41:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:41:18.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1648 2024-09-16T12:41:18.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1649 2024-09-16T12:41:18.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:19.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1650 2024-09-16T12:41:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:19.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:41:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:19.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:41:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:41:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:41:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:41:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:41:19.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1654 2024-09-16T12:41:20.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1655 2024-09-16T12:41:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:41:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:41:20.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:41:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:41:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:20.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:41:20.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:41:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1656 2024-09-16T12:41:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:41:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:41:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:41:22.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1661 2024-09-16T12:41:22.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:41:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:41:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:41:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1660 2024-09-16T12:41:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1662 2024-09-16T12:41:22.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:22.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:41:22.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:22.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:22.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:41:22.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:41:22.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:22.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:22.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:22.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1666 2024-09-16T12:41:22.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:41:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:41:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:41:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:41:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:41:23.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:41:23.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1669 2024-09-16T12:41:23.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1668 2024-09-16T12:41:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:23.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1670 2024-09-16T12:41:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:41:24.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:41:24.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:41:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1675 2024-09-16T12:41:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1674 2024-09-16T12:41:24.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:41:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:41:25.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:41:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:41:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:25.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:41:25.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:41:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:41:25.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:41:25.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:41:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1678 2024-09-16T12:41:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:41:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:41:25.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:41:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:26.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1682 2024-09-16T12:41:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:41:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:41:26.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:41:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:26.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:41:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:41:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:41:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1680 2024-09-16T12:41:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:26.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1681 2024-09-16T12:41:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:41:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:41:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:41:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:26.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1686 2024-09-16T12:41:27.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:27.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:41:27.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:27.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:27.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:41:27.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:41:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:27.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:27.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:41:27.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:27.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:27.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:41:27.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:41:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:41:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:41:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:41:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1689 2024-09-16T12:41:28.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1688 2024-09-16T12:41:28.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1690 2024-09-16T12:41:28.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:28.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:41:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:41:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:41:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1694 2024-09-16T12:41:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:41:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:41:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:41:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:41:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:41:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:41:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1695 2024-09-16T12:41:29.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:29.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:41:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:41:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:41:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1698 2024-09-16T12:41:30.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1702 2024-09-16T12:41:31.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1700 2024-09-16T12:41:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1701 2024-09-16T12:41:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:41:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:41:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:41:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:41:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:41:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:41:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:41:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:41:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:41:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:32.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1705 2024-09-16T12:41:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:41:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:41:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:41:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:41:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:41:32.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:41:33.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1709 2024-09-16T12:41:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:41:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:41:33.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:41:33.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1708 2024-09-16T12:41:33.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1710 2024-09-16T12:41:33.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:33.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:33.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:34.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1714 2024-09-16T12:41:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1715 2024-09-16T12:41:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:34.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:41:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:41:34.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:41:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:41:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:41:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:41:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:41:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:41:34.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:41:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:41:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:35.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:41:35.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:41:35.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1718 2024-09-16T12:41:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:36.891 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:41:36.882+0000 7ff4f6946700 -1 reset not still connected to 0x557576938dd0 2024-09-16T12:41:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:41:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:36.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:41:36.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:41:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:41:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:41:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:41:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1720 2024-09-16T12:41:36.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1722 2024-09-16T12:41:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:41:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:41:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:41:36.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1721 2024-09-16T12:41:36.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:36.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:41:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:41:37.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:41:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:41:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:41:37.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:41:37.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1723 2024-09-16T12:41:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1730 2024-09-16T12:41:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:41:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:41:38.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:41:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1728 2024-09-16T12:41:38.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1727 2024-09-16T12:41:38.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:38.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:38.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1735 2024-09-16T12:41:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:39.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:41:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:41:39.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:41:39.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1734 2024-09-16T12:41:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:41:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:39.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:41:39.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:41:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:41:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:41:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:41:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:41:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:41:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:41:40.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1737 2024-09-16T12:41:40.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:41:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:41:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:41:41.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1740 2024-09-16T12:41:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:41:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:41:41.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:41:41.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1741 2024-09-16T12:41:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:41:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:41.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:41:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:41:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1742 2024-09-16T12:41:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:41.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:42.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1746 2024-09-16T12:41:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:42.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:41:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:41:42.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:41:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:41:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:42.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:41:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:41:43.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1748 2024-09-16T12:41:43.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1749 2024-09-16T12:41:43.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1750 2024-09-16T12:41:43.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:43.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:43.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:41:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:41:43.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:41:44.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1755 2024-09-16T12:41:44.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1754 2024-09-16T12:41:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:41:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:41:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:41:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:41:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:41:44.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:41:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:41:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:41:44.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:41:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1758 2024-09-16T12:41:45.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:41:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:41:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:41:45.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:46.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:41:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:41:46.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:41:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1761 2024-09-16T12:41:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:41:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:41:46.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:41:46.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1762 2024-09-16T12:41:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:41:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:41:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:41:46.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1760 2024-09-16T12:41:46.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:46.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:47.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1766 2024-09-16T12:41:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:41:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:41:47.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:41:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:41:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:47.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:41:47.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:41:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1768 2024-09-16T12:41:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:41:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:48.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:41:48.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:41:48.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1769 2024-09-16T12:41:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1770 2024-09-16T12:41:48.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:48.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1775 2024-09-16T12:41:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:41:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:41:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:41:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1774 2024-09-16T12:41:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:41:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:41:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:41:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:41:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:41:49.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:41:49.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:50.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:41:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:41:50.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:41:50.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1777 2024-09-16T12:41:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:41:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:41:51.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:41:51.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1782 2024-09-16T12:41:51.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:41:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:41:51.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:41:51.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1781 2024-09-16T12:41:51.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:51.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:41:51.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:51.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:51.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:41:51.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:41:51.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1780 2024-09-16T12:41:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:52.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1786 2024-09-16T12:41:52.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:41:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:41:52.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:41:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:41:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:41:52.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:41:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:53.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:41:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:41:53.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:41:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1787 2024-09-16T12:41:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1789 2024-09-16T12:41:53.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:53.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:53.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1790 2024-09-16T12:41:53.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1794 2024-09-16T12:41:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:41:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:41:54.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:41:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:41:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:54.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:41:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:41:54.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1795 2024-09-16T12:41:54.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:54.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:41:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:41:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:41:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:41:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:41:55.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:41:55.916 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:41:55.906+0000 7f4071c2f700 -1 reset not still connected to 0x5635a8e416c0 2024-09-16T12:41:55.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1798 2024-09-16T12:41:55.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:41:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:41:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:41:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:41:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:41:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:41:56.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1800 2024-09-16T12:41:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1801 2024-09-16T12:41:56.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:56.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:56.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:41:56.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:56.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:56.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:41:56.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:41:56.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1802 2024-09-16T12:41:56.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:56.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:57.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1803 2024-09-16T12:41:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:41:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:57.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:41:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:41:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:41:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:41:57.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:41:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:41:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:41:58.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:41:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1809 2024-09-16T12:41:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1808 2024-09-16T12:41:58.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1810 2024-09-16T12:41:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:58.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:58.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:41:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:41:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:41:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1814 2024-09-16T12:41:59.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1815 2024-09-16T12:41:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:41:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:41:59.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:41:59.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:41:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:41:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:41:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:41:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:41:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:41:59.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:42:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:42:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:42:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:42:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1816 2024-09-16T12:42:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1820 2024-09-16T12:42:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:42:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:01.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:42:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:42:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:42:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:42:01.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:42:01.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1822 2024-09-16T12:42:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:42:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:01.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:42:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:42:01.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1821 2024-09-16T12:42:01.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:01.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:01.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:02.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1826 2024-09-16T12:42:02.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:42:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:42:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:42:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:42:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:42:03.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:42:03.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1829 2024-09-16T12:42:03.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:03.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1830 2024-09-16T12:42:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1828 2024-09-16T12:42:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:42:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:42:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:42:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1834 2024-09-16T12:42:04.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1835 2024-09-16T12:42:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:42:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:42:04.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:42:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:42:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:42:04.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:42:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:05.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:42:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:42:05.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:42:05.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1838 2024-09-16T12:42:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:42:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:42:05.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:42:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1842 2024-09-16T12:42:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:42:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:42:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:42:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1841 2024-09-16T12:42:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1840 2024-09-16T12:42:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:42:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:42:06.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:42:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:42:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:42:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:42:06.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:06.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1846 2024-09-16T12:42:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:42:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:42:07.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:42:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:42:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:07.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:42:07.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:42:07.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:42:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:42:08.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:42:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1849 2024-09-16T12:42:08.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1847 2024-09-16T12:42:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1850 2024-09-16T12:42:08.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:08.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1854 2024-09-16T12:42:09.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1855 2024-09-16T12:42:09.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:09.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:09.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:42:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:42:09.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:42:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:42:09.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:42:09.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:42:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:42:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:42:09.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:42:10.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:42:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:42:10.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:42:10.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1856 2024-09-16T12:42:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:11.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1861 2024-09-16T12:42:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:42:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:42:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:42:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:42:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:42:11.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:42:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1860 2024-09-16T12:42:11.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:42:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:42:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:42:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:11.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:12.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1866 2024-09-16T12:42:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:12.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1863 2024-09-16T12:42:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:12.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:42:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:42:12.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:42:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:42:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:42:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:42:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:42:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:42:13.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:42:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1871 2024-09-16T12:42:13.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1870 2024-09-16T12:42:13.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1869 2024-09-16T12:42:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:13.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:13.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:14.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:42:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:42:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:42:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1875 2024-09-16T12:42:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:42:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:42:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:42:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1876 2024-09-16T12:42:14.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:14.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:15.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:15.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:42:15.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:15.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:15.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:42:15.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:42:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:42:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:42:15.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:42:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1879 2024-09-16T12:42:15.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:15.947 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:42:15.934+0000 7f61b1855700 -1 reset not still connected to 0x557b39beb040 2024-09-16T12:42:16.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1881 2024-09-16T12:42:16.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1882 2024-09-16T12:42:16.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:42:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:16.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:42:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:42:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:42:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:42:16.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:42:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:42:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:42:16.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:42:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:17.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1883 2024-09-16T12:42:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:42:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:42:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:42:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:42:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:42:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:42:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1885 2024-09-16T12:42:17.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:17.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:18.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:42:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:42:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:42:18.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1890 2024-09-16T12:42:18.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1889 2024-09-16T12:42:18.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1891 2024-09-16T12:42:18.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:18.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:42:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:42:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:42:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1895 2024-09-16T12:42:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:42:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:42:19.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:42:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1896 2024-09-16T12:42:19.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:42:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:42:20.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:42:20.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1897 2024-09-16T12:42:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:42:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:42:20.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:42:20.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:42:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:42:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:42:21.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1901 2024-09-16T12:42:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1902 2024-09-16T12:42:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:42:21.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 51218 2024-09-16T12:42:21.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:21.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:42:21.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:42:21.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:42:22.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1905 2024-09-16T12:42:22.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1906 2024-09-16T12:42:22.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:22.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:42:22.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:42:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:42:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:42:22.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 51353 2024-09-16T12:42:23.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1908 2024-09-16T12:42:23.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1910 2024-09-16T12:42:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:42:23.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 51488 2024-09-16T12:42:24.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1913 2024-09-16T12:42:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:42:24.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 51662 2024-09-16T12:42:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:42:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:42:24.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:42:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:42:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:42:25.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:42:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:42:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:42:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:42:26.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1915 2024-09-16T12:42:26.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:42:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:42:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:42:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:42:26.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 51817 2024-09-16T12:42:26.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:42:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 51974 2024-09-16T12:42:27.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1918 2024-09-16T12:42:27.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1917 2024-09-16T12:42:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:27.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:28.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1921 2024-09-16T12:42:28.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:42:29.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 52151 2024-09-16T12:42:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:42:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:42:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:42:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:42:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:42:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:42:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:42:30.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 52327 2024-09-16T12:42:31.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:42:31.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 52595 2024-09-16T12:42:32.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1923 2024-09-16T12:42:32.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:42:35.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T12:42:35.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:42:35.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 51085 2024-09-16T12:43:28.719 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:43:28.709+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T12:43:28.722 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:43:28.713+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T12:44:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:44:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:44:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:44:52.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 51219 2024-09-16T12:44:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:44:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:44:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:44:54.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 51354 2024-09-16T12:44:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:44:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:44:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:44:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 51489 2024-09-16T12:44:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:44:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:44:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:44:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 51663 2024-09-16T12:44:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:44:58.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:44:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:44:58.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 51818 2024-09-16T12:44:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:44:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:44:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:44:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 51975 2024-09-16T12:45:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:45:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:45:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:45:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 52152 2024-09-16T12:45:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:45:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:45:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:45:02.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 52328 2024-09-16T12:45:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:45:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:45:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:45:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 52596 2024-09-16T12:45:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:45:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:45:06.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T12:45:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:45:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:45:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:45:06.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 1 2024-09-16T12:45:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:45:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd0 2024-09-16T12:45:06.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary1 2024-09-16T12:45:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary1 2024-09-16T12:45:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:45:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:45:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary1 2024-09-16T12:45:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 2 2024-09-16T12:45:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd1 2024-09-16T12:45:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary2 2024-09-16T12:45:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:45:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary2 2024-09-16T12:45:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:45:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary2 2024-09-16T12:45:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:45:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 3 2024-09-16T12:45:06.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:45:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd2 2024-09-16T12:45:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:45:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary3 2024-09-16T12:45:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary3 2024-09-16T12:45:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:45:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary3 2024-09-16T12:45:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:45:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:45:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 5 2024-09-16T12:45:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:45:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 4 2024-09-16T12:45:06.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:45:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd4 2024-09-16T12:45:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary5 2024-09-16T12:45:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary5 2024-09-16T12:45:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:45:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary5 2024-09-16T12:45:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd3 2024-09-16T12:45:06.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary4 2024-09-16T12:45:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary4 2024-09-16T12:45:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:45:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:45:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary4 2024-09-16T12:45:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:45:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 7 2024-09-16T12:45:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:45:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd6 2024-09-16T12:45:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary7 2024-09-16T12:45:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary7 2024-09-16T12:45:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:45:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:45:06.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary7 2024-09-16T12:45:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 8 2024-09-16T12:45:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 6 2024-09-16T12:45:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd7 2024-09-16T12:45:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:45:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary8 2024-09-16T12:45:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary8 2024-09-16T12:45:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:45:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary8 2024-09-16T12:45:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:45:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 9 2024-09-16T12:45:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd5 2024-09-16T12:45:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary6 2024-09-16T12:45:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary6 2024-09-16T12:45:06.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:45:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:45:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary6 2024-09-16T12:45:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:45:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 58551 2024-09-16T12:45:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd8 2024-09-16T12:45:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary9 2024-09-16T12:45:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary9 2024-09-16T12:45:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:45:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary9 2024-09-16T12:45:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 10 2024-09-16T12:45:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd9 2024-09-16T12:45:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary10 2024-09-16T12:45:06.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary10 2024-09-16T12:45:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:45:06.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary10 2024-09-16T12:45:06.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:45:06.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd6 2024-09-16T12:45:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd4 2024-09-16T12:45:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:45:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd2 2024-09-16T12:45:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:45:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:45:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd1 2024-09-16T12:45:06.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:45:06.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd9 2024-09-16T12:45:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd7 2024-09-16T12:45:06.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:45:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:45:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd0 2024-09-16T12:45:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd3 2024-09-16T12:45:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:45:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:45:06.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd8 2024-09-16T12:45:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd5 2024-09-16T12:45:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:46:02.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=2795d8857bcf0fc2b525e27990f8b27a 2024-09-16T12:46:02.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary9 2024-09-16T12:46:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=894d1d573411f254f33053b2e980a926 2024-09-16T12:46:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary6 2024-09-16T12:46:02.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=9c1d6c4a26c5b98c1cbbb6cdfe4a9365 2024-09-16T12:46:02.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary2 2024-09-16T12:46:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=36b4a1b45ab885a2dff98e6b59b4f4e2 2024-09-16T12:46:02.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary4 2024-09-16T12:46:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=e7b459db9581b80efb84b5019a53dea4 2024-09-16T12:46:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary10 2024-09-16T12:46:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=c0fbe204183af6c92166221528767319 2024-09-16T12:46:02.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary3 2024-09-16T12:46:03.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=b3be42e04dbe5d85169ea3ca4b2be3c7 2024-09-16T12:46:03.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary1 2024-09-16T12:46:03.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=11e94cb0c7f70deddd7d8e9ea82d5d2c 2024-09-16T12:46:03.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary7 2024-09-16T12:46:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=3c61ab33a218bd6d981d9f871c9532e0 2024-09-16T12:46:04.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary5 2024-09-16T12:46:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=d9ab5273d2877c9e096e74c274cff5d0 2024-09-16T12:46:04.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary8 2024-09-16T12:46:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-16T12:46:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:04.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:46:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-16T12:46:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-16T12:46:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:04.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:46:04.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-16T12:46:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-16T12:46:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:04.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:46:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-16T12:46:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-16T12:46:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:46:04.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-16T12:46:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-16T12:46:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:46:04.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-16T12:46:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-16T12:46:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:46:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-16T12:46:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-16T12:46:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:46:04.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-16T12:46:04.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-16T12:46:04.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:04.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:04.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:46:04.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-16T12:46:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-16T12:46:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:46:04.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-16T12:46:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-16T12:46:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:46:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-16T12:46:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:46:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:46:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:46:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:46:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:46:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:46:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:46:05.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:46:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-16T12:46:05.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:46:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:46:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-16T12:46:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:46:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:46:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-16T12:46:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:46:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:46:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-16T12:46:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:46:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:46:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-16T12:46:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:46:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:46:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-16T12:46:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-16T12:46:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:46:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:46:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:46:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:46:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-16T12:46:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:46:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:46:05.983 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:46:05.973+0000 7f4071c2f700 -1 reset not still connected to 0x5635a8fa6270 2024-09-16T12:46:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:46:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:46:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-16T12:46:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:46:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:06.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:46:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-16T12:46:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:46:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:46:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-16T12:46:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-16T12:46:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-16T12:46:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-16T12:46:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:07.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:46:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:46:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:46:07.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:46:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-16T12:46:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:07.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:46:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-16T12:46:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-16T12:46:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:07.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:07.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:07.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:46:07.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:46:07.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:07.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:07.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:07.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:07.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:07.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:07.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:07.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:07.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:07.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T12:46:07.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:07.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T12:46:07.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:07.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T12:46:07.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:07.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T12:46:07.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:07.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T12:46:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T12:46:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-16T12:46:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:46:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T12:46:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:07.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T12:46:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T12:46:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a80f552e-4c2a-48a2-8fa0-3670fc11d7e8 2024-09-16T12:46:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:46:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:46:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:03 2024-09-16T12:46:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:46:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:46:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3459072.0,"last_snapshot_bytes":6782976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490545,"remote_snapshot_timestamp":1726490545,"replay_state":"idle"} 2024-09-16T12:46:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:45:53' 2024-09-16T12:46:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T12:46:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 18c1291b-9eb8-46a9-82b4-f0d012b14821 2024-09-16T12:46:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:46:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:46:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:03 2024-09-16T12:46:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:07.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:46:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:46:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:06' 2024-09-16T12:46:07.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T12:46:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: f057b092-1b46-4f3e-b5ba-d6581e9d4b6c 2024-09-16T12:46:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:46:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:46:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:07.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:03 2024-09-16T12:46:07.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:07.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:46:07.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:46:07.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":929792.0,"last_snapshot_bytes":180224,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490547,"remote_snapshot_timestamp":1726490547,"replay_state":"idle"} 2024-09-16T12:46:07.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:45:53' 2024-09-16T12:46:07.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T12:46:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 04708389-b988-49ba-9b70-98157102cb71 2024-09-16T12:46:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:46:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:46:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:03 2024-09-16T12:46:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:46:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:46:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:06' 2024-09-16T12:46:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T12:46:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 9e488d88-4f84-437a-a54a-0803bfdd4803 2024-09-16T12:46:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:46:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:46:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:03 2024-09-16T12:46:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:46:07.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:46:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:06' 2024-09-16T12:46:07.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T12:46:07.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: cfa45128-953f-406f-a7c1-a2354ab18676 2024-09-16T12:46:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:46:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:46:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:03 2024-09-16T12:46:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:46:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:46:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2621440.0,"last_snapshot_bytes":229376,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490551,"remote_snapshot_timestamp":1726490551,"replay_state":"idle"} 2024-09-16T12:46:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:45:53' 2024-09-16T12:46:07.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T12:46:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 00da4166-a0f4-4e0d-a0cb-e3ce5afbffa1 2024-09-16T12:46:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:46:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:46:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:03 2024-09-16T12:46:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:46:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:46:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2150400.0,"last_snapshot_bytes":2551808,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490537,"remote_snapshot_timestamp":1726490537,"replay_state":"idle"} 2024-09-16T12:46:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:45:53' 2024-09-16T12:46:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T12:46:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: fa948dbe-a7fa-43eb-b157-1a11a03be35d 2024-09-16T12:46:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:46:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:46:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:03 2024-09-16T12:46:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:46:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:46:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:06' 2024-09-16T12:46:07.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:46:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:46:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:46:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:46:07.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:46:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:46:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:46:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:46:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:46:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:46:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:46:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:46:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:46:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:46:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:46:07.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:46:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-16T12:46:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:46:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:07.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:07.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T12:46:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-16T12:46:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:46:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:07.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T12:46:08.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T12:46:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 387ce0f1-7b85-4979-8cef-4635c6f65150 2024-09-16T12:46:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:46:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:46:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:03 2024-09-16T12:46:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:46:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:46:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:07' 2024-09-16T12:46:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T12:46:08.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 18ee13c8-2f19-45e6-885e-456ffd58cc67 2024-09-16T12:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:03 2024-09-16T12:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:07' 2024-09-16T12:46:08.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:46:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:46:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:46:08.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:08.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:46:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-16T12:46:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-16T12:46:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-16T12:46:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-16T12:46:09.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-16T12:46:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:46:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:46:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-16T12:46:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-16T12:46:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:46:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:46:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:46:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:46:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:46:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-16T12:46:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:46:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:09.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T12:46:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T12:46:09.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T12:46:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T12:46:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T12:46:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T12:46:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T12:46:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T12:46:09.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T12:46:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 9e488d88-4f84-437a-a54a-0803bfdd4803 2024-09-16T12:46:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08 2024-09-16T12:46:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:46:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08' 2024-09-16T12:46:09.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:46:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:46:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T12:46:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a80f552e-4c2a-48a2-8fa0-3670fc11d7e8 2024-09-16T12:46:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08 2024-09-16T12:46:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:46:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08' 2024-09-16T12:46:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:46:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:46:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:09.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:46:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:46:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:46:09.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:46:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:46:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:09.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:09.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:46:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:46:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:46:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:46:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:46:09.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:09.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T12:46:09.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 04708389-b988-49ba-9b70-98157102cb71 2024-09-16T12:46:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08 2024-09-16T12:46:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:46:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08' 2024-09-16T12:46:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T12:46:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: f057b092-1b46-4f3e-b5ba-d6581e9d4b6c 2024-09-16T12:46:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08 2024-09-16T12:46:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:46:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08' 2024-09-16T12:46:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:46:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:46:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T12:46:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: cfa45128-953f-406f-a7c1-a2354ab18676 2024-09-16T12:46:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08 2024-09-16T12:46:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:46:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08' 2024-09-16T12:46:09.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:46:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:46:09.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:46:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T12:46:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 18c1291b-9eb8-46a9-82b4-f0d012b14821 2024-09-16T12:46:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08 2024-09-16T12:46:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:46:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08' 2024-09-16T12:46:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:46:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:46:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:46:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:46:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:46:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:46:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:46:09.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:46:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:46:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:46:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:46:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:09.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:46:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:46:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T12:46:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: fa948dbe-a7fa-43eb-b157-1a11a03be35d 2024-09-16T12:46:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08 2024-09-16T12:46:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:46:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08' 2024-09-16T12:46:09.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:46:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T12:46:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 00da4166-a0f4-4e0d-a0cb-e3ce5afbffa1 2024-09-16T12:46:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08 2024-09-16T12:46:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:46:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08' 2024-09-16T12:46:09.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:46:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:09.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:46:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:46:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:09.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:46:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:46:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:46:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:46:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:46:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:46:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:46:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:46:09.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:46:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:46:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:46:09.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:46:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:46:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:09.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:46:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:46:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:09.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:46:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:46:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:46:09.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:09.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:46:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:46:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-16 12:46:08' 2024-09-16T12:46:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-16 12:46:08' 2024-09-16T12:46:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:46:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:46:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:46:08' 2024-09-16T12:46:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:46:08' 2024-09-16T12:46:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-16 12:46:08' 2024-09-16T12:46:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:09.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:46:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:46:08' 2024-09-16T12:46:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-16 12:46:08' 2024-09-16T12:46:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-16 12:46:08' 2024-09-16T12:46:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-16 12:46:08' 2024-09-16T12:46:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:46:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:46:08' 2024-09-16T12:46:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:46:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:46:08' 2024-09-16T12:46:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:46:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:46:08' 2024-09-16T12:46:09.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-16 12:46:08' 2024-09-16T12:46:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:46:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:46:08' 2024-09-16T12:46:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-16 12:46:08' 2024-09-16T12:46:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:46:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:46:08' 2024-09-16T12:46:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:09.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary304708389-b988-49ba-9b70-98157102cb71up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08' 2024-09-16T12:46:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:46:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T12:46:09.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary304708389-b988-49ba-9b70-98157102cb71up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08' 2024-09-16T12:46:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:46:09.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T12:46:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary304708389-b988-49ba-9b70-98157102cb71up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08' 2024-09-16T12:46:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:46:09.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T12:46:09.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:09.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T12:46:09.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:09.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:09.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T12:46:09.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T12:46:09.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:46:09.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:46:09.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:09.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:46:09.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:09.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:46:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:46:09.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-16T12:46:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary304708389-b988-49ba-9b70-98157102cb71up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2<+ local pool=mirror 2024-09-16T12:46:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:46:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mirror_uuids>c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08' 2024-09-16T12:46:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:46:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:46:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:46:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T12:46:09.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:09.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-16T12:46:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:46:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:46:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-16T12:46:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:46:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:46:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary304708389-b988-49ba-9b70-98157102cb71up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08' 2024-09-16T12:46:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:46:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T12:46:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary304708389-b988-49ba-9b70-98157102cb71up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08' 2024-09-16T12:46:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:46:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T12:46:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary304708389-b988-49ba-9b70-98157102cb71up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08' 2024-09-16T12:46:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:46:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T12:46:09.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T12:46:09.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary304708389-b988-49ba-9b70-98157102cb71up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08' 2024-09-16T12:46:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:46:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T12:46:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:46:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:46:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:09.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T12:46:09.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-16T12:46:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:46:09.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:46:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T12:46:09.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:46:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T12:46:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:46:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T12:46:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:46:09.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:09.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:09.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:46:09.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:09.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:09.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-16T12:46:09.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:46:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:46:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-16T12:46:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:46:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:46:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:46:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:46:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:09.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-16T12:46:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:46:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:46:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:46:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:46:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-16T12:46:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:09.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:46:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:46:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-16T12:46:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-16T12:46:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:46:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:46:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T12:46:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:10.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T12:46:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T12:46:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 387ce0f1-7b85-4979-8cef-4635c6f65150 2024-09-16T12:46:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08 2024-09-16T12:46:10.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:46:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T12:46:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 18ee13c8-2f19-45e6-885e-456ffd58cc67 2024-09-16T12:46:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08 2024-09-16T12:46:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:46:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08' 2024-09-16T12:46:10.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08' 2024-09-16T12:46:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:10.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:46:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:46:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:46:10.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:46:10.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:46:10.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:46:10.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:46:10.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:46:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:46:10.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:46:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:10.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:10.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:46:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:46:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:46:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:46:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-16 12:46:08' 2024-09-16T12:46:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:10.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:46:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:46:08' 2024-09-16T12:46:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-16 12:46:08' 2024-09-16T12:46:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:46:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:46:10.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:46:08' 2024-09-16T12:46:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:10.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary304708389-b988-49ba-9b70-98157102cb71up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08' 2024-09-16T12:46:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:46:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T12:46:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T12:46:10.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:46:10.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:10.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:46:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:10.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:10.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:10.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-16T12:46:10.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:10.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:46:10.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:10.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:46:10.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary304708389-b988-49ba-9b70-98157102cb71up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:46:08cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:46:08' 2024-09-16T12:46:10.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:46:10.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T12:46:10.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:10.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T12:46:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:46:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:46:10.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-16T12:46:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:46:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:46:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:46:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-16T12:46:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:46:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:10.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-16T12:46:10.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-16T12:46:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:46:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:10.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-16T12:46:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-16T12:46:10.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:46:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:10.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-16T12:46:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-16T12:46:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:46:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:10.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-16T12:46:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-16T12:46:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:46:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:10.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-16T12:46:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-16T12:46:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:46:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:10.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-16T12:46:10.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-16T12:46:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:46:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:10.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-16T12:46:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-16T12:46:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:46:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:10.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-16T12:46:10.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T12:46:10.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 9e488d88-4f84-437a-a54a-0803bfdd4803 2024-09-16T12:46:10.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:10.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:10.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:10.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08 2024-09-16T12:46:10.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:10.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:46:10.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:10.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:10.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08' 2024-09-16T12:46:10.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:46:10.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:46:10.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:46:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:46:10.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:46:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:10.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:10.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:46:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:46:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T12:46:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 04708389-b988-49ba-9b70-98157102cb71 2024-09-16T12:46:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08 2024-09-16T12:46:10.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:46:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08' 2024-09-16T12:46:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T12:46:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 18c1291b-9eb8-46a9-82b4-f0d012b14821 2024-09-16T12:46:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08 2024-09-16T12:46:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:46:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08' 2024-09-16T12:46:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T12:46:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: cfa45128-953f-406f-a7c1-a2354ab18676 2024-09-16T12:46:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08 2024-09-16T12:46:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:46:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08' 2024-09-16T12:46:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-16 12:46:08' 2024-09-16T12:46:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:46:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:46:08' 2024-09-16T12:46:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:10.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:46:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T12:46:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: fa948dbe-a7fa-43eb-b157-1a11a03be35d 2024-09-16T12:46:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08 2024-09-16T12:46:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:46:10.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08' 2024-09-16T12:46:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:46:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:46:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T12:46:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: f057b092-1b46-4f3e-b5ba-d6581e9d4b6c 2024-09-16T12:46:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08 2024-09-16T12:46:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:46:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08' 2024-09-16T12:46:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:46:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:46:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:46:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:46:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:46:10.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:46:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:46:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:46:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:46:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:46:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:46:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:46:10.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:46:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:46:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:46:10.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:46:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:46:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:46:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:46:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:46:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:46:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:46:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:46:10.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:46:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:10.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:46:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T12:46:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a80f552e-4c2a-48a2-8fa0-3670fc11d7e8 2024-09-16T12:46:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08 2024-09-16T12:46:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:46:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08' 2024-09-16T12:46:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:46:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T12:46:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 00da4166-a0f4-4e0d-a0cb-e3ce5afbffa1 2024-09-16T12:46:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:10.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08 2024-09-16T12:46:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:46:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08' 2024-09-16T12:46:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:10.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:46:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-16 12:46:08' 2024-09-16T12:46:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:46:10.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:46:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:10.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:46:08' 2024-09-16T12:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:10.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:46:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:46:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-16 12:46:08' 2024-09-16T12:46:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:46:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:46:08' 2024-09-16T12:46:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:46:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:10.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:46:10.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-16 12:46:08' 2024-09-16T12:46:10.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:10.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:46:10.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:46:08' 2024-09-16T12:46:10.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:10.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:46:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:46:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:46:10.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:46:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:46:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:46:10.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:10.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:10.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:46:10.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:46:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-16 12:46:08' 2024-09-16T12:46:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:46:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:46:08' 2024-09-16T12:46:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:10.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-16 12:46:08' 2024-09-16T12:46:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:46:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:46:08' 2024-09-16T12:46:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:10.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:46:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:46:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-16 12:46:08' 2024-09-16T12:46:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:46:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:46:08' 2024-09-16T12:46:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-16 12:46:08' 2024-09-16T12:46:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:46:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:46:08' 2024-09-16T12:46:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary304708389-b988-49ba-9b70-98157102cb71up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08' 2024-09-16T12:46:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:46:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T12:46:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T12:46:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:46:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:11.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:46:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:11.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-16T12:46:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:46:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:46:11.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-16T12:46:11.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary304708389-b988-49ba-9b70-98157102cb71up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08' 2024-09-16T12:46:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:46:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T12:46:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T12:46:11.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:46:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:11.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:46:11.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-16T12:46:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:11.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:46:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:46:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-16T12:46:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary304708389-b988-49ba-9b70-98157102cb71up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08' 2024-09-16T12:46:11.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:46:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary304708389-b988-49ba-9b70-98157102cb71up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08' 2024-09-16T12:46:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:46:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T12:46:11.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T12:46:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary304708389-b988-49ba-9b70-98157102cb71up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08' 2024-09-16T12:46:11.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:46:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary304708389-b988-49ba-9b70-98157102cb71up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08' 2024-09-16T12:46:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:46:11.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T12:46:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary304708389-b988-49ba-9b70-98157102cb71up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08' 2024-09-16T12:46:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:46:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T12:46:11.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T12:46:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T12:46:11.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:11.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T12:46:11.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary304708389-b988-49ba-9b70-98157102cb71up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08' 2024-09-16T12:46:11.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:46:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T12:46:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:11.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:46:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T12:46:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T12:46:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:46:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:46:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:46:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T12:46:11.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-16T12:46:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:46:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:46:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-16T12:46:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:46:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:46:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T12:46:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-16T12:46:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:46:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:46:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-16T12:46:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:46:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:46:11.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:11.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-16T12:46:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:46:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:46:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-16T12:46:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:46:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-16T12:46:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:46:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-16T12:46:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:46:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:46:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-16T12:46:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-16T12:46:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:46:11.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:46:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:11.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-16T12:46:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:46:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:46:11.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-16T12:46:11.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:46:11.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-16T12:46:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:46:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:46:11.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-16T12:46:11.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T12:46:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 18ee13c8-2f19-45e6-885e-456ffd58cc67 2024-09-16T12:46:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08 2024-09-16T12:46:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:46:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08' 2024-09-16T12:46:11.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:46:11.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:46:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:46:11.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:46:11.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:46:11.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:11.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:11.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:46:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:46:11.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-16 12:46:08' 2024-09-16T12:46:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:46:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:46:08' 2024-09-16T12:46:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:11.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary304708389-b988-49ba-9b70-98157102cb71up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08' 2024-09-16T12:46:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:46:11.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T12:46:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-16T12:46:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:46:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:46:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:46:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-16T12:46:11.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:11.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T12:46:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:46:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:11.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:46:11.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-16T12:46:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:46:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:46:11.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-16T12:46:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T12:46:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 387ce0f1-7b85-4979-8cef-4635c6f65150 2024-09-16T12:46:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:46:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08 2024-09-16T12:46:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:46:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:46:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:46:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:46:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:46:08' 2024-09-16T12:46:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:46:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:46:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:46:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:46:11.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:46:11.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:11.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:46:11.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:11.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:46:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:46:11.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:46:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:46:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-16 12:46:08' 2024-09-16T12:46:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:46:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:46:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:46:08' 2024-09-16T12:46:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:46:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:46:11.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:46:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary304708389-b988-49ba-9b70-98157102cb71up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:46:08cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:46:08' 2024-09-16T12:46:11.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:46:11.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T12:46:11.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:46:08' 2024-09-16T12:46:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T12:46:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:46:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:46:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:46:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:46:11.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:46:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-16T12:46:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:46:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:46:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:46:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:46:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-16T12:46:12.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:46:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:46:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:46:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:46:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:46:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:46:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:46:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:46:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:46:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:46:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:46:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary2 2024-09-16T12:46:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary5 2024-09-16T12:46:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:46:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:46:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary6 2024-09-16T12:46:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:46:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary4 2024-09-16T12:46:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:46:12.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary8 2024-09-16T12:46:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:46:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:46:12.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary10 2024-09-16T12:46:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary7 2024-09-16T12:46:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:46:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary9 2024-09-16T12:46:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:46:12.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary1 2024-09-16T12:46:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:46:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary3 2024-09-16T12:46:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd0 2024-09-16T12:46:12.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:46:12.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd0 2024-09-16T12:46:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd1 2024-09-16T12:46:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd1 2024-09-16T12:46:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:46:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd2 2024-09-16T12:46:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:46:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd2 2024-09-16T12:46:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd3 2024-09-16T12:46:12.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:46:12.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd3 2024-09-16T12:46:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd4 2024-09-16T12:46:12.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:46:12.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd4 2024-09-16T12:46:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd5 2024-09-16T12:46:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:46:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd5 2024-09-16T12:46:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd6 2024-09-16T12:46:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:46:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd6 2024-09-16T12:46:12.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd7 2024-09-16T12:46:12.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:46:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd7 2024-09-16T12:46:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd8 2024-09-16T12:46:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:46:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd8 2024-09-16T12:46:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd9 2024-09-16T12:46:12.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:46:12.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd9 2024-09-16T12:47:48.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=36b4a1b45ab885a2dff98e6b59b4f4e2 2024-09-16T12:47:48.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd3 2024-09-16T12:47:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=2795d8857bcf0fc2b525e27990f8b27a 2024-09-16T12:47:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd6 2024-09-16T12:47:49.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=b3be42e04dbe5d85169ea3ca4b2be3c7 2024-09-16T12:47:49.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd9 2024-09-16T12:47:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=894d1d573411f254f33053b2e980a926 2024-09-16T12:47:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd5 2024-09-16T12:47:49.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=d9ab5273d2877c9e096e74c274cff5d0 2024-09-16T12:47:49.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd2 2024-09-16T12:47:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=3c61ab33a218bd6d981d9f871c9532e0 2024-09-16T12:47:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd4 2024-09-16T12:47:49.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=e7b459db9581b80efb84b5019a53dea4 2024-09-16T12:47:49.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-09-16T12:47:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 36b4a1b45ab885a2dff98e6b59b4f4e2 != \3\6\b\4\a\1\b\4\5\a\b\8\8\5\a\2\d\f\f\9\8\e\6\b\5\9\b\4\f\4\e\2 ]] 2024-09-16T12:47:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=c0fbe204183af6c92166221528767319 2024-09-16T12:47:50.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd8 2024-09-16T12:47:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=9c1d6c4a26c5b98c1cbbb6cdfe4a9365 2024-09-16T12:47:50.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd1 2024-09-16T12:47:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 2795d8857bcf0fc2b525e27990f8b27a != \2\7\9\5\d\8\8\5\7\b\c\f\0\f\c\2\b\5\2\5\e\2\7\9\9\0\f\8\b\2\7\a ]] 2024-09-16T12:47:50.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=11e94cb0c7f70deddd7d8e9ea82d5d2c 2024-09-16T12:47:50.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd7 2024-09-16T12:47:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ b3be42e04dbe5d85169ea3ca4b2be3c7 != \b\3\b\e\4\2\e\0\4\d\b\e\5\d\8\5\1\6\9\e\a\3\c\a\4\b\2\b\e\3\c\7 ]] 2024-09-16T12:47:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:47:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 58552 2024-09-16T12:47:50.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ d9ab5273d2877c9e096e74c274cff5d0 != \d\9\a\b\5\2\7\3\d\2\8\7\7\c\9\e\0\9\6\e\7\4\c\2\7\4\c\f\f\5\d\0 ]] 2024-09-16T12:47:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 894d1d573411f254f33053b2e980a926 != \8\9\4\d\1\d\5\7\3\4\1\1\f\2\5\4\f\3\3\0\5\3\b\2\e\9\8\0\a\9\2\6 ]] 2024-09-16T12:47:50.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ e7b459db9581b80efb84b5019a53dea4 != \e\7\b\4\5\9\d\b\9\5\8\1\b\8\0\e\f\b\8\4\b\5\0\1\9\a\5\3\d\e\a\4 ]] 2024-09-16T12:47:50.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 3c61ab33a218bd6d981d9f871c9532e0 != \3\c\6\1\a\b\3\3\a\2\1\8\b\d\6\d\9\8\1\d\9\f\8\7\1\c\9\5\3\2\e\0 ]] 2024-09-16T12:47:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 9c1d6c4a26c5b98c1cbbb6cdfe4a9365 != \9\c\1\d\6\c\4\a\2\6\c\5\b\9\8\c\1\c\b\b\b\6\c\d\f\e\4\a\9\3\6\5 ]] 2024-09-16T12:47:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:47:50.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 58554 2024-09-16T12:47:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ c0fbe204183af6c92166221528767319 != \c\0\f\b\e\2\0\4\1\8\3\a\f\6\c\9\2\1\6\6\2\2\1\5\2\8\7\6\7\3\1\9 ]] 2024-09-16T12:47:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:47:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 58555 2024-09-16T12:47:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:47:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 58557 2024-09-16T12:47:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:47:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 58559 2024-09-16T12:47:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:47:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 58560 2024-09-16T12:47:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 11e94cb0c7f70deddd7d8e9ea82d5d2c != \1\1\e\9\4\c\b\0\c\7\f\7\0\d\e\d\d\d\7\d\8\e\9\e\a\8\2\d\5\d\2\c ]] 2024-09-16T12:47:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:47:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 58563 2024-09-16T12:47:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:47:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 58565 2024-09-16T12:47:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:47:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 58567 2024-09-16T12:47:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T12:47:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:47:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary1 2024-09-16T12:47:50.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-16T12:47:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:47:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:47:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:47:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:47:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:47:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:47:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:47:50.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:47:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-16T12:47:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:47:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:47:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:47:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:47:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:47:51.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:47:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:47:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:47:51.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T12:47:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T12:47:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 00da4166-a0f4-4e0d-a0cb-e3ce5afbffa1 2024-09-16T12:47:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:47:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70176768.0,"last_snapshot_bytes":70176768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"} 2024-09-16T12:47:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:47:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:47:23 2024-09-16T12:47:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:47:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:47:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:47:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:47:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:47:48' 2024-09-16T12:47:51.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:47:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:47:51.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:47:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70176768.0,"last_snapshot_bytes":70176768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"} 2024-09-16T12:47:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:47:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:47:51.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:47:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:47:51.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:47:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:47:51.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:47:51.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:47:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:47:23' 2024-09-16T12:47:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-16 12:47:23' 2024-09-16T12:47:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:47:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:47:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:47:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:47:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:47:23' 2024-09-16T12:47:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:47:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:47:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:47:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary100da4166-a0f4-4e0d-a0cb-e3ce5afbffa1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70176768.0,"last_snapshot_bytes":70176768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:48image-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62111744.0,"last_snapshot_bytes":62111744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:48image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72294400.0,"last_snapshot_bytes":72294400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:48image-primary304708389-b988-49ba-9b70-98157102cb71up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65372160.0,"last_snapshot_bytes":65372160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:48image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72335360.0,"last_snapshot_bytes":72335360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:48image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65392640.0,"last_snapshot_bytes":65392640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:48image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70250496.0,"last_snapshot_bytes":70250496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:48image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60452864.0,"last_snapshot_bytes":60452864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:48image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:48image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72437760.0,"last_snapshot_bytes":72437760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:48' 2024-09-16T12:47:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:47:52.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T12:47:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:47:23' 2024-09-16T12:47:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T12:47:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:47:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:47:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:47:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:47:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:47:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:47:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-16T12:47:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:47:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:47:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:47:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-16T12:47:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-16T12:47:54.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:47:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-16T12:47:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:47:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:47:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:47:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:47:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:47:54.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary1 2024-09-16T12:47:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:47:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary1 2024-09-16T12:47:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:47:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary2 2024-09-16T12:47:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-16T12:47:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:47:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:47:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:47:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:47:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:47:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:47:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:47:54.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:47:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-16T12:47:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:47:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:47:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:47:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:47:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:47:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:47:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:47:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:47:55.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T12:47:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T12:47:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: fa948dbe-a7fa-43eb-b157-1a11a03be35d 2024-09-16T12:47:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:47:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72294400.0,"last_snapshot_bytes":72294400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"} 2024-09-16T12:47:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:47:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:47:53 2024-09-16T12:47:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:47:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:47:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:47:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:47:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:47:53' 2024-09-16T12:47:55.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:47:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:47:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:47:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72294400.0,"last_snapshot_bytes":72294400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"} 2024-09-16T12:47:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:47:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:47:55.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:47:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:47:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:47:55.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:47:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:47:55.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:47:55.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:47:53' 2024-09-16T12:47:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-16 12:47:53' 2024-09-16T12:47:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:47:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:47:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:47:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:47:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:47:53' 2024-09-16T12:47:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:47:55.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:47:55.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:47:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK942544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62111744.0,"last_snapshot_bytes":62111744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:53image-primary2fa948dbe-a7fa-43eb-b157-1a11a03be35dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72294400.0,"last_snapshot_bytes":72294400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:53image-primary304708389-b988-49ba-9b70-98157102cb71up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65372160.0,"last_snapshot_bytes":65372160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:53image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72335360.0,"last_snapshot_bytes":72335360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:53image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65392640.0,"last_snapshot_bytes":65392640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:53image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70250496.0,"last_snapshot_bytes":70250496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:53image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60452864.0,"last_snapshot_bytes":60452864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:53image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:53image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72437760.0,"last_snapshot_bytes":72437760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:53' 2024-09-16T12:47:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:47:55.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T12:47:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:47:53' 2024-09-16T12:47:55.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T12:47:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:47:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:47:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:47:55.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:47:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:47:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:47:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-16T12:47:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:47:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:47:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:47:55.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-16T12:47:55.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-16T12:47:57.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:47:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-16T12:47:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:47:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:47:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:47:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:47:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:47:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary2 2024-09-16T12:47:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:47:57.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary2 2024-09-16T12:47:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:47:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary3 2024-09-16T12:47:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-16T12:47:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:47:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:47:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:47:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:47:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:47:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:47:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:47:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:47:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-16T12:47:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:47:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:47:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:47:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:47:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:47:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:47:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:47:58.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:47:58.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T12:47:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T12:47:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 04708389-b988-49ba-9b70-98157102cb71 2024-09-16T12:47:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:47:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65372160.0,"last_snapshot_bytes":65372160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"} 2024-09-16T12:47:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:47:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:47:53 2024-09-16T12:47:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:47:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:47:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:47:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:47:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:47:53' 2024-09-16T12:47:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:47:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:47:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:47:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65372160.0,"last_snapshot_bytes":65372160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"} 2024-09-16T12:47:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:47:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:47:58.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:47:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:47:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:47:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:47:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:47:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:47:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:47:53' 2024-09-16T12:47:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-16 12:47:53' 2024-09-16T12:47:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:47:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:47:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:47:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:47:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:47:53' 2024-09-16T12:47:58.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:47:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:47:58.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:47:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK842544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62111744.0,"last_snapshot_bytes":62111744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:53image-primary304708389-b988-49ba-9b70-98157102cb71up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65372160.0,"last_snapshot_bytes":65372160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:53image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72335360.0,"last_snapshot_bytes":72335360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:53image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65392640.0,"last_snapshot_bytes":65392640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:53image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70250496.0,"last_snapshot_bytes":70250496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:53image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60452864.0,"last_snapshot_bytes":60452864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:53image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:53image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72437760.0,"last_snapshot_bytes":72437760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:53' 2024-09-16T12:47:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:47:58.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T12:47:58.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:47:53' 2024-09-16T12:47:58.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T12:47:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:47:58.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:47:58.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:47:58.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:47:58.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:47:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:47:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-16T12:47:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:47:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:47:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:47:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-16T12:47:58.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-16T12:47:59.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:47:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-16T12:47:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:47:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:47:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:47:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:47:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:47:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary3 2024-09-16T12:47:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:47:59.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary3 2024-09-16T12:47:59.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:47:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary4 2024-09-16T12:47:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-16T12:47:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:47:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:47:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:47:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:47:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:47:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:47:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:47:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:48:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-16T12:48:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:48:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:48:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:48:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:48:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:48:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T12:48:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T12:48:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 9e488d88-4f84-437a-a54a-0803bfdd4803 2024-09-16T12:48:00.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:48:00.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72335360.0,"last_snapshot_bytes":72335360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"} 2024-09-16T12:48:00.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:48:00.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:47:53 2024-09-16T12:48:00.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:48:00.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:48:00.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:48:00.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:48:00.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:47:58' 2024-09-16T12:48:00.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:48:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:48:00.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:48:00.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72335360.0,"last_snapshot_bytes":72335360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"} 2024-09-16T12:48:00.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:48:00.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:48:00.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:48:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:48:00.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:48:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:48:00.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:48:00.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:48:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:47:53' 2024-09-16T12:48:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-16 12:47:53' 2024-09-16T12:48:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:48:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:47:53' 2024-09-16T12:48:00.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:48:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:48:00.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:48:01.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK742544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62111744.0,"last_snapshot_bytes":62111744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:58image-primary49e488d88-4f84-437a-a54a-0803bfdd4803up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72335360.0,"last_snapshot_bytes":72335360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:58image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65392640.0,"last_snapshot_bytes":65392640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:58image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70250496.0,"last_snapshot_bytes":70250496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:58image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60452864.0,"last_snapshot_bytes":60452864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:58image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:58image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72437760.0,"last_snapshot_bytes":72437760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:47:58' 2024-09-16T12:48:01.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:48:01.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T12:48:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:47:53' 2024-09-16T12:48:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T12:48:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:48:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:48:01.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:48:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:48:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-16T12:48:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:48:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:48:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-16T12:48:01.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-16T12:48:02.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:48:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-16T12:48:02.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:48:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:48:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:48:02.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary4 2024-09-16T12:48:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary4 2024-09-16T12:48:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:48:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary5 2024-09-16T12:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-16T12:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:48:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-16T12:48:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:48:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:48:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:48:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:48:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:48:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T12:48:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T12:48:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 18ee13c8-2f19-45e6-885e-456ffd58cc67 2024-09-16T12:48:03.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:48:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65392640.0,"last_snapshot_bytes":65392640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"} 2024-09-16T12:48:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:48:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:47:53 2024-09-16T12:48:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:48:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:48:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:48:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:48:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:48:03' 2024-09-16T12:48:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:48:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:48:03.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:48:03.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65392640.0,"last_snapshot_bytes":65392640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"} 2024-09-16T12:48:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:48:03.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:48:03.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:48:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:48:03.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:48:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:48:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:48:03.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:48:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:47:53' 2024-09-16T12:48:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-16 12:47:53' 2024-09-16T12:48:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:48:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:47:53' 2024-09-16T12:48:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:48:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:48:03.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:48:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK642544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62111744.0,"last_snapshot_bytes":62111744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:03image-primary518ee13c8-2f19-45e6-885e-456ffd58cc67up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65392640.0,"last_snapshot_bytes":65392640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:03image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70250496.0,"last_snapshot_bytes":70250496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:03image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60452864.0,"last_snapshot_bytes":60452864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:03image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:03image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72437760.0,"last_snapshot_bytes":72437760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:03' 2024-09-16T12:48:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:48:03.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T12:48:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:47:53' 2024-09-16T12:48:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T12:48:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:48:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:48:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:48:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-16T12:48:03.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:48:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:48:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-16T12:48:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-16T12:48:05.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:48:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-16T12:48:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:48:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:48:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:48:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary5 2024-09-16T12:48:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:05.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary5 2024-09-16T12:48:05.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:48:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary6 2024-09-16T12:48:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-16T12:48:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:48:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:48:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:48:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:48:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:48:06.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-16T12:48:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:48:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:48:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:48:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:48:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:48:06.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T12:48:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T12:48:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 18c1291b-9eb8-46a9-82b4-f0d012b14821 2024-09-16T12:48:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:48:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70250496.0,"last_snapshot_bytes":70250496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"} 2024-09-16T12:48:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:48:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:47:53 2024-09-16T12:48:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:48:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:48:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:48:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:48:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:48:03' 2024-09-16T12:48:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:48:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:48:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:48:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70250496.0,"last_snapshot_bytes":70250496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"} 2024-09-16T12:48:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:48:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:48:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:48:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:48:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:48:06.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:48:06.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:48:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:48:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:47:53' 2024-09-16T12:48:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-16 12:47:53' 2024-09-16T12:48:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:06.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:48:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:47:53' 2024-09-16T12:48:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:48:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:48:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:48:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK542544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62111744.0,"last_snapshot_bytes":62111744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:03image-primary618c1291b-9eb8-46a9-82b4-f0d012b14821up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70250496.0,"last_snapshot_bytes":70250496,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:03image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60452864.0,"last_snapshot_bytes":60452864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:03image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:03image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72437760.0,"last_snapshot_bytes":72437760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:03' 2024-09-16T12:48:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:48:06.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T12:48:06.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:47:53' 2024-09-16T12:48:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T12:48:06.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:48:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:48:06.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:48:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:48:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-16T12:48:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:48:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:48:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-16T12:48:06.914 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:48:06.904+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T12:48:06.918 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:48:06.908+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T12:48:06.918 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:48:06.908+0000 7f61b1855700 -1 reset not still connected to 0x557b36ee2270 2024-09-16T12:48:06.918 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:48:06.908+0000 7f61b1855700 -1 reset not still connected to 0x557b4215f520 2024-09-16T12:48:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-16T12:48:08.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:48:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-16T12:48:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:48:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:48:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:48:08.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary6 2024-09-16T12:48:08.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:08.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary6 2024-09-16T12:48:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:48:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary7 2024-09-16T12:48:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-16T12:48:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:48:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:48:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:48:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:48:08.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:48:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-16T12:48:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:48:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:48:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:48:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:48:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:48:09.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T12:48:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T12:48:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a80f552e-4c2a-48a2-8fa0-3670fc11d7e8 2024-09-16T12:48:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:48:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60452864.0,"last_snapshot_bytes":60452864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"} 2024-09-16T12:48:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:48:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:47:53 2024-09-16T12:48:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:48:09.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:48:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:48:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:48:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:48:08' 2024-09-16T12:48:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:48:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:48:09.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:48:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60452864.0,"last_snapshot_bytes":60452864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"} 2024-09-16T12:48:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:48:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:48:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:48:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:48:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:48:09.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:48:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:48:09.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:48:09.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:47:53' 2024-09-16T12:48:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-16 12:47:53' 2024-09-16T12:48:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:48:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:47:53' 2024-09-16T12:48:09.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:48:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:48:09.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:48:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK442544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62111744.0,"last_snapshot_bytes":62111744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:08image-primary7a80f552e-4c2a-48a2-8fa0-3670fc11d7e8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60452864.0,"last_snapshot_bytes":60452864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72437760.0,"last_snapshot_bytes":72437760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:08' 2024-09-16T12:48:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:48:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T12:48:10.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:47:53' 2024-09-16T12:48:10.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T12:48:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:48:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:48:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:48:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:48:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-16T12:48:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:48:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:48:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-16T12:48:10.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-16T12:48:11.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:48:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-16T12:48:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:11.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:11.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:48:11.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:48:11.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:48:11.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary7 2024-09-16T12:48:11.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:11.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary7 2024-09-16T12:48:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:48:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary8 2024-09-16T12:48:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-16T12:48:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:48:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:48:11.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:48:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:48:11.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:48:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-16T12:48:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:48:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:12.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:48:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:48:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:48:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:48:12.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T12:48:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T12:48:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 387ce0f1-7b85-4979-8cef-4635c6f65150 2024-09-16T12:48:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:48:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"} 2024-09-16T12:48:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:48:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:47:53 2024-09-16T12:48:12.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:48:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:48:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:48:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:48:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:48:08' 2024-09-16T12:48:12.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:48:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:48:12.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:48:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"} 2024-09-16T12:48:12.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:48:12.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:48:12.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:48:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:48:12.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:48:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:48:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:48:12.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:48:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:47:53' 2024-09-16T12:48:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-16 12:47:53' 2024-09-16T12:48:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:48:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:47:53' 2024-09-16T12:48:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:48:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:48:12.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:48:13.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK342544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62111744.0,"last_snapshot_bytes":62111744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:08image-primary8387ce0f1-7b85-4979-8cef-4635c6f65150up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:08image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72437760.0,"last_snapshot_bytes":72437760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:08' 2024-09-16T12:48:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:48:13.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T12:48:13.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:47:53' 2024-09-16T12:48:13.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T12:48:13.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:48:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:48:13.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:48:13.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:48:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-16T12:48:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:48:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:48:13.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-16T12:48:13.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-16T12:48:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:48:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-16T12:48:14.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:48:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:48:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:48:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary8 2024-09-16T12:48:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:14.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary8 2024-09-16T12:48:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:48:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary9 2024-09-16T12:48:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-16T12:48:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:48:14.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:48:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:48:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:48:14.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:48:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-16T12:48:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:48:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:48:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:48:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:48:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:48:15.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T12:48:15.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T12:48:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: f057b092-1b46-4f3e-b5ba-d6581e9d4b6c 2024-09-16T12:48:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:48:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72437760.0,"last_snapshot_bytes":72437760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"} 2024-09-16T12:48:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:48:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:47:53 2024-09-16T12:48:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:48:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:48:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:48:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:48:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:48:13' 2024-09-16T12:48:15.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:48:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:48:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:48:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72437760.0,"last_snapshot_bytes":72437760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"} 2024-09-16T12:48:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:48:15.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:48:15.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:48:15.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:48:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:48:15.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:48:15.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:48:15.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:48:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:47:53' 2024-09-16T12:48:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-16 12:47:53' 2024-09-16T12:48:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:48:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:47:53' 2024-09-16T12:48:15.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:48:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:48:15.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:48:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK242544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62111744.0,"last_snapshot_bytes":62111744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:13image-primary9f057b092-1b46-4f3e-b5ba-d6581e9d4b6cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72437760.0,"last_snapshot_bytes":72437760,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:13' 2024-09-16T12:48:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:48:15.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T12:48:15.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:47:53' 2024-09-16T12:48:15.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T12:48:15.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:48:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:48:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:48:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:48:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:15.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-16T12:48:15.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:48:15.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:15.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:48:15.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-16T12:48:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-16T12:48:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:48:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-16T12:48:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:17.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:48:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:48:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:48:17.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary9 2024-09-16T12:48:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary9 2024-09-16T12:48:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:48:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary10 2024-09-16T12:48:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-16T12:48:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:48:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:48:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:48:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:48:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:48:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-16T12:48:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:48:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:48:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:48:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:48:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:48:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T12:48:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T12:48:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: cfa45128-953f-406f-a7c1-a2354ab18676 2024-09-16T12:48:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:48:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62111744.0,"last_snapshot_bytes":62111744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"} 2024-09-16T12:48:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:48:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:47:53 2024-09-16T12:48:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:48:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:48:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:48:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:48:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:48:18' 2024-09-16T12:48:19.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:48:19.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:48:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:48:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62111744.0,"last_snapshot_bytes":62111744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"} 2024-09-16T12:48:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:48:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:48:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:48:19.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:48:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:48:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:48:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:48:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:48:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:47:53' 2024-09-16T12:48:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-16 12:47:53' 2024-09-16T12:48:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:48:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:48:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:47:53' 2024-09-16T12:48:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:48:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:48:19.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:48:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK142544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10cfa45128-953f-406f-a7c1-a2354ab18676up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62111744.0,"last_snapshot_bytes":62111744,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726490772,"remote_snapshot_timestamp":1726490772,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:48:18' 2024-09-16T12:48:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:48:19.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T12:48:19.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:47:53' 2024-09-16T12:48:19.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T12:48:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:48:19.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:48:19.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:48:19.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:48:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:19.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-16T12:48:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:48:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:48:19.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-16T12:48:19.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-16T12:48:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:48:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-16T12:48:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:48:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:48:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:48:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary10 2024-09-16T12:48:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:48:21.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary10 2024-09-16T12:48:21.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..10} 2024-09-16T12:48:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS=() 2024-09-16T12:48:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS=() 2024-09-16T12:48:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS=() 2024-09-16T12:48:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS=() 2024-09-16T12:48:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ RET=0 2024-09-16T12:48:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:48:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary1 2024-09-16T12:48:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary1 2024-09-16T12:48:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-16T12:48:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:48:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:48:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:48:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-16T12:48:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:48:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:48:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:48:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:48:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-16T12:48:21.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:48:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-16T12:48:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:48:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:48:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-16T12:48:22.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:48:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-16T12:48:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary1': 2024-09-16T12:48:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:48:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:48:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:48:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1e9f93d1f3c0 2024-09-16T12:48:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1e9f93d1f3c0 2024-09-16T12:48:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:48:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:48:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:48:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:48:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:48:21 2024 2024-09-16T12:48:23.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:48:21 2024 2024-09-16T12:48:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:48:21 2024 2024-09-16T12:48:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:48:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:48:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: c1088e42-e596-4d28-9923-42d6045d5571 2024-09-16T12:48:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:48:23.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:48:23.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary1 2024-09-16T12:48:23.184 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:48:23.176+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T12:48:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd0 2024-09-16T12:48:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:48:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd0 2024-09-16T12:48:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:48:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:48:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:48:23.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 575e2368-5029-42be-8b8e-f72a8eb76619 2024-09-16T12:48:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:48:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:48:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:48:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:48:23.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:48:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:48:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:48:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:48:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary1 2024-09-16T12:48:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd0 test-primary1 2024-09-16T12:48:23.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:48:23.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary1 2024-09-16T12:48:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:48:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:48:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:48:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary2 2024-09-16T12:48:23.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary2 2024-09-16T12:48:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-16T12:48:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-16T12:48:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:48:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:48:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:48:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-16T12:48:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:48:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:48:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:48:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:48:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary1 2024-09-16T12:48:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:48:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-16T12:48:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:48:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:48:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:48:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary1 2024-09-16T12:48:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-16T12:48:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:48:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-16T12:48:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:48:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:48:23.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-16T12:48:24.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:48:24.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-16T12:48:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1980 2024-09-16T12:48:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:25.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:48:25.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-16T12:48:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary2': 2024-09-16T12:48:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:48:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:48:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:48:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1ead23d2577b 2024-09-16T12:48:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1ead23d2577b 2024-09-16T12:48:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:48:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:48:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:48:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:48:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:48:23 2024 2024-09-16T12:48:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:48:23 2024 2024-09-16T12:48:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:48:23 2024 2024-09-16T12:48:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:48:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:48:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 8d0e5cc3-e3f8-4977-9e72-a3c1492e4e85 2024-09-16T12:48:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:48:25.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:48:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary2 2024-09-16T12:48:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd1 2024-09-16T12:48:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:48:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd1 2024-09-16T12:48:25.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:48:25.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:48:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:48:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 60100df6-ada3-400e-b467-82111fd15baf 2024-09-16T12:48:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:48:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:48:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:48:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:48:25.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:48:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:48:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:48:25.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:48:25.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary2 2024-09-16T12:48:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd1 test-primary2 2024-09-16T12:48:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:48:25.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary2 2024-09-16T12:48:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:48:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:48:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-16T12:48:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:48:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary3 2024-09-16T12:48:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary3 2024-09-16T12:48:25.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-16T12:48:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary2 2024-09-16T12:48:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:48:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:48:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:48:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:48:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-16T12:48:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary2 2024-09-16T12:48:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-16T12:48:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:48:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:48:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:48:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:48:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:48:25.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-16T12:48:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:48:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:48:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:48:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-16T12:48:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:25.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:48:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:48:25.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-16T12:48:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:48:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-16T12:48:26.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1984 2024-09-16T12:48:26.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:48:26.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:26.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-16T12:48:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary3': 2024-09-16T12:48:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:48:26.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:48:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:48:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1ebc3864e35b 2024-09-16T12:48:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1ebc3864e35b 2024-09-16T12:48:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:48:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:48:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:48:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:48:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:48:25 2024 2024-09-16T12:48:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:48:25 2024 2024-09-16T12:48:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:48:25 2024 2024-09-16T12:48:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:48:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:48:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: fec07e41-ee3f-4c53-baf9-b248297f6077 2024-09-16T12:48:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:48:26.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:48:26.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary3 2024-09-16T12:48:26.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd2 2024-09-16T12:48:26.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:48:26.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd2 2024-09-16T12:48:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:48:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:48:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:48:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 7968b3b2-1340-47da-a553-54364471bc0f 2024-09-16T12:48:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:48:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:48:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:48:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:48:26.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:48:26.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:48:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:48:26.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:48:26.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary3 2024-09-16T12:48:26.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd2 test-primary3 2024-09-16T12:48:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:48:26.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary3 2024-09-16T12:48:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:48:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-16T12:48:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary3 2024-09-16T12:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary4 2024-09-16T12:48:26.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary4 2024-09-16T12:48:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-16T12:48:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary3 2024-09-16T12:48:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:48:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-16T12:48:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:48:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:48:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-16T12:48:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:48:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:48:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:48:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:48:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:26.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-16T12:48:26.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:48:26.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-16T12:48:26.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:26.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:26.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:48:26.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:48:26.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-16T12:48:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:48:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-16T12:48:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:48:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:48:27.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:48:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1987 2024-09-16T12:48:27.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:48:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-16T12:48:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary4': 2024-09-16T12:48:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:48:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:48:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:48:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1eccfac7ff70 2024-09-16T12:48:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1eccfac7ff70 2024-09-16T12:48:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:48:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:48:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:48:27.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:48:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:48:26 2024 2024-09-16T12:48:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:48:26 2024 2024-09-16T12:48:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:48:26 2024 2024-09-16T12:48:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:48:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:48:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: dd6269fd-f848-44f5-b3e7-af6943c54ccb 2024-09-16T12:48:27.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:48:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:48:27.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary4 2024-09-16T12:48:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd3 2024-09-16T12:48:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:48:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd3 2024-09-16T12:48:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:48:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:48:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:48:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 838bb626-d671-4c49-a76f-1f29e0c28878 2024-09-16T12:48:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:48:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:48:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:48:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:48:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:48:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:48:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:48:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:48:27.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary4 2024-09-16T12:48:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd3 test-primary4 2024-09-16T12:48:27.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:48:27.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary4 2024-09-16T12:48:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:48:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:48:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-16T12:48:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:48:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary5 2024-09-16T12:48:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary5 2024-09-16T12:48:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-16T12:48:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:48:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:48:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:27.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:48:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:48:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:48:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-16T12:48:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:48:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:48:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:48:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:48:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:48:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:48:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary4 2024-09-16T12:48:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-16T12:48:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary4 2024-09-16T12:48:27.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-16T12:48:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:48:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-16T12:48:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:48:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:48:28.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-16T12:48:28.118 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:48:28.108+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0ea0 2024-09-16T12:48:28.119 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:48:28.108+0000 7f61b1855700 -1 reset not still connected to 0x557b44ed84e0 2024-09-16T12:48:28.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:48:28.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-16T12:48:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1989 2024-09-16T12:48:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:48:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:48:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:48:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1990 2024-09-16T12:48:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:48:29.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-16T12:48:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary5': 2024-09-16T12:48:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:48:29.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1edd16287f41 2024-09-16T12:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1edd16287f41 2024-09-16T12:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:48:28 2024 2024-09-16T12:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:48:28 2024 2024-09-16T12:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:48:28 2024 2024-09-16T12:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: aedce2c3-b936-41c9-95f9-3b2be5e43d04 2024-09-16T12:48:29.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:48:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:48:29.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary5 2024-09-16T12:48:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd4 2024-09-16T12:48:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:48:29.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd4 2024-09-16T12:48:29.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:48:29.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:48:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:48:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 5d49b66c-f962-494e-9fc0-6359a0debb7a 2024-09-16T12:48:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:48:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:48:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:48:29.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:48:29.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:48:29.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:48:29.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:48:29.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:48:29.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary5 2024-09-16T12:48:29.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd4 test-primary5 2024-09-16T12:48:29.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:48:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary5 2024-09-16T12:48:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:48:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-16T12:48:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:29.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:48:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:48:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:48:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:48:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary6 2024-09-16T12:48:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary5 2024-09-16T12:48:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary6 2024-09-16T12:48:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:48:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-16T12:48:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:48:29.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary5 2024-09-16T12:48:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:48:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:48:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-16T12:48:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:48:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-16T12:48:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:48:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:48:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:48:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:48:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-16T12:48:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:48:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-16T12:48:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:48:30.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:48:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-16T12:48:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:48:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-16T12:48:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:48:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:48:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:48:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1993 2024-09-16T12:48:31.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1999 2024-09-16T12:48:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:31.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:48:31.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-16T12:48:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary6': 2024-09-16T12:48:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:48:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:48:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:48:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1eee3677e225 2024-09-16T12:48:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1eee3677e225 2024-09-16T12:48:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:48:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:48:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:48:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:48:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:48:30 2024 2024-09-16T12:48:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:48:30 2024 2024-09-16T12:48:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:48:30 2024 2024-09-16T12:48:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:48:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:48:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: e1674f59-5785-4885-8027-0915586882c9 2024-09-16T12:48:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:48:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:48:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary6 2024-09-16T12:48:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd5 2024-09-16T12:48:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:48:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd5 2024-09-16T12:48:31.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:48:31.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:48:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:48:31.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 8f5c6d85-da72-461f-adcc-cb05904ae1ce 2024-09-16T12:48:31.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:48:31.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:48:31.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:48:31.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:48:31.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:48:31.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:48:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:48:31.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:48:31.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary6 2024-09-16T12:48:31.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd5 test-primary6 2024-09-16T12:48:31.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:48:31.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary6 2024-09-16T12:48:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:48:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-16T12:48:31.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:48:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:48:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:48:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary7 2024-09-16T12:48:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary7 2024-09-16T12:48:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:48:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-16T12:48:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:48:31.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:48:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:48:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:48:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:48:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-16T12:48:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:31.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:48:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:48:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:48:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:48:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-16T12:48:31.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary6 2024-09-16T12:48:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary6 2024-09-16T12:48:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-16T12:48:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:48:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-16T12:48:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:32.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:48:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:48:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-16T12:48:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:48:32.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-16T12:48:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:48:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:48:32.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:48:32.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 1998 2024-09-16T12:48:32.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:32.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2002 2024-09-16T12:48:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:48:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:48:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:48:32.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:32.658 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:48:32.648+0000 7f4071c2f700 -1 reset not still connected to 0x563591bfbba0 2024-09-16T12:48:32.658 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:48:32.648+0000 7f4071c2f700 -1 reset not still connected to 0x563591d7d040 2024-09-16T12:48:32.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:48:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-16T12:48:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary7': 2024-09-16T12:48:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:48:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:48:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:48:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1f022681132f 2024-09-16T12:48:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1f022681132f 2024-09-16T12:48:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:48:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:48:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:48:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:48:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:48:31 2024 2024-09-16T12:48:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:48:31 2024 2024-09-16T12:48:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:48:31 2024 2024-09-16T12:48:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:48:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:48:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: afa1056f-626c-48ac-b10e-3d158ee2f57a 2024-09-16T12:48:33.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:48:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:48:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary7 2024-09-16T12:48:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd6 2024-09-16T12:48:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:48:33.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd6 2024-09-16T12:48:33.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:48:33.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:48:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:48:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 3d99c2cd-ab07-4ab7-941f-4526ab56662b 2024-09-16T12:48:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:48:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:48:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:48:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:48:33.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:48:33.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:48:33.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:48:33.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:48:33.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary7 2024-09-16T12:48:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd6 test-primary7 2024-09-16T12:48:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:48:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary7 2024-09-16T12:48:33.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:48:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:48:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:48:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary8 2024-09-16T12:48:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary8 2024-09-16T12:48:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-16T12:48:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-16T12:48:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:48:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:48:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:48:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-16T12:48:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:48:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:48:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:48:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:48:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:48:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:48:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-16T12:48:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:48:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:48:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary7 2024-09-16T12:48:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary7 2024-09-16T12:48:33.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-16T12:48:33.996 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:48:33.988+0000 7f61b1855700 -1 reset not still connected to 0x557b229664e0 2024-09-16T12:48:33.997 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:48:33.988+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T12:48:33.997 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:48:33.988+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T12:48:33.998 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:48:33.988+0000 7f61b1855700 -1 reset not still connected to 0x557b4c026ea0 2024-09-16T12:48:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:48:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-16T12:48:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:48:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:48:34.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:48:34.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2004 2024-09-16T12:48:34.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2005 2024-09-16T12:48:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:48:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:48:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:48:34.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:48:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-16T12:48:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:48:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:48:34.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-16T12:48:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:48:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:48:35.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:48:35.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2008 2024-09-16T12:48:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:48:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:48:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:48:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:48:35.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:35.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-16T12:48:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary8': 2024-09-16T12:48:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:48:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:48:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:48:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1f157faf7150 2024-09-16T12:48:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1f157faf7150 2024-09-16T12:48:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:48:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:48:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:48:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:48:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:48:34 2024 2024-09-16T12:48:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:48:34 2024 2024-09-16T12:48:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:48:34 2024 2024-09-16T12:48:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:48:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:48:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 729e14b6-bf4a-4677-9904-ec7f7484589d 2024-09-16T12:48:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:48:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:48:35.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary8 2024-09-16T12:48:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd7 2024-09-16T12:48:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:48:35.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd7 2024-09-16T12:48:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:48:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:48:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:48:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 2c8983c8-337f-42e7-995a-d73b71a517f9 2024-09-16T12:48:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:48:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:48:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:48:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:48:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:48:35.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:48:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:48:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:48:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary8 2024-09-16T12:48:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd7 test-primary8 2024-09-16T12:48:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:48:35.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary8 2024-09-16T12:48:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:48:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-16T12:48:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:48:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:48:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:48:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary9 2024-09-16T12:48:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary9 2024-09-16T12:48:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:48:35.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-16T12:48:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary8 2024-09-16T12:48:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:48:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:48:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:48:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:48:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:48:35.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-16T12:48:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:48:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:48:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary8 2024-09-16T12:48:35.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:48:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:48:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-16T12:48:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-16T12:48:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:48:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-16T12:48:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:48:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:48:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-16T12:48:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:48:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-16T12:48:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2009 2024-09-16T12:48:36.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2010 2024-09-16T12:48:36.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:36.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2016 2024-09-16T12:48:36.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:48:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:48:37.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:48:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:48:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:48:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:48:37.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2014 2024-09-16T12:48:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:37.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2015 2024-09-16T12:48:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:37.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:48:37.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-16T12:48:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary9': 2024-09-16T12:48:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:48:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:48:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:48:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1f2d3a7ae41f 2024-09-16T12:48:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1f2d3a7ae41f 2024-09-16T12:48:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:48:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:48:37.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:48:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:48:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:48:36 2024 2024-09-16T12:48:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:48:36 2024 2024-09-16T12:48:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:48:36 2024 2024-09-16T12:48:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:48:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:48:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 4734d245-46e2-4286-95a3-3cc562d928fa 2024-09-16T12:48:37.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:48:37.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:48:37.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary9 2024-09-16T12:48:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd8 2024-09-16T12:48:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:48:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd8 2024-09-16T12:48:37.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:48:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:48:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:48:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 86403d37-7f36-4fe3-9b4f-acac35dcda05 2024-09-16T12:48:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:48:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:48:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:48:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:48:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:48:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:48:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:48:37.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:48:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary9 2024-09-16T12:48:37.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd8 test-primary9 2024-09-16T12:48:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:48:38.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary9 2024-09-16T12:48:38.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:48:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-16T12:48:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:48:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:48:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:48:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:48:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary9 2024-09-16T12:48:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary10 2024-09-16T12:48:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:38.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary10 2024-09-16T12:48:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:48:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:48:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-16T12:48:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary9 2024-09-16T12:48:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-16T12:48:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:38.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:48:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:48:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:48:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-16T12:48:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:48:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:38.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:48:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:48:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:48:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:48:38.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-16T12:48:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:48:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-16T12:48:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:48:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:48:38.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-16T12:48:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:48:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:48:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:48:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:48:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-16T12:48:39.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:48:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:48:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:48:39.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:48:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:48:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:48:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2020 2024-09-16T12:48:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:39.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:48:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:48:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:48:39.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2021 2024-09-16T12:48:39.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:39.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2022 2024-09-16T12:48:39.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:48:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-16T12:48:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary10': 2024-09-16T12:48:39.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:48:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:48:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:48:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 1f41d92e2703 2024-09-16T12:48:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.1f41d92e2703 2024-09-16T12:48:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:48:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:48:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:48:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:48:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:48:38 2024 2024-09-16T12:48:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:48:38 2024 2024-09-16T12:48:39.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:48:38 2024 2024-09-16T12:48:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:48:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:48:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: c82934ef-5cb6-459e-a80f-ec8e46fbc42a 2024-09-16T12:48:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:48:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:48:39.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary10 2024-09-16T12:48:39.476 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:48:39.465+0000 7f74def64700 -1 reset not still connected to 0x559455bdc680 2024-09-16T12:48:39.477 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:48:39.465+0000 7f74def64700 -1 reset not still connected to 0x5594601c80d0 2024-09-16T12:48:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd9 2024-09-16T12:48:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:48:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd9 2024-09-16T12:48:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:48:39.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:48:39.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:48:39.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 3bbb034b-37c4-4c56-8775-6dfe912b1754 2024-09-16T12:48:39.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:48:39.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:48:39.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:48:39.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:48:39.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:48:39.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:48:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:48:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:48:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary10 2024-09-16T12:48:39.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd9 test-primary10 2024-09-16T12:48:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:48:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary10 2024-09-16T12:48:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:48:40.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-16T12:48:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:48:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:48:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:48:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:48:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:48:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary10 2024-09-16T12:48:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:48:40.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 62756 2024-09-16T12:48:40.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary10 2024-09-16T12:48:40.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-16T12:48:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2026 2024-09-16T12:48:40.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:48:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-16T12:48:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:40.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:48:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:48:40.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:48:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:48:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:48:40.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:48:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2030 2024-09-16T12:48:41.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2027 2024-09-16T12:48:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2028 2024-09-16T12:48:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:41.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:41.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2031 2024-09-16T12:48:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:42.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2035 2024-09-16T12:48:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:48:42.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:48:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:48:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:48:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:48:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:48:42.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2036 2024-09-16T12:48:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:48:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:48:42.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:48:42.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:48:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:48:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:48:43.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2040 2024-09-16T12:48:43.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2041 2024-09-16T12:48:43.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2039 2024-09-16T12:48:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:43.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:43.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:48:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:48:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:48:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:48:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:44.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:48:44.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:48:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:48:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:48:44.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:48:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:48:44.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:48:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:48:44.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2045 2024-09-16T12:48:44.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:48:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:48:45.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:48:45.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:45.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:48:45.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:45.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:45.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:48:45.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:48:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2050 2024-09-16T12:48:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2049 2024-09-16T12:48:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2047 2024-09-16T12:48:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2048 2024-09-16T12:48:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:45.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:46.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:48:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:48:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:48:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:46.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:48:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:48:46.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:48:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2055 2024-09-16T12:48:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:48:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:48:46.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:48:46.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2056 2024-09-16T12:48:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:48:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:48:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:48:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2061 2024-09-16T12:48:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2059 2024-09-16T12:48:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2060 2024-09-16T12:48:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:48:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:48:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:48:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:48:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:48:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:48:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:48:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:48:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:48:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:48:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:48:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:48:49.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2065 2024-09-16T12:48:49.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:48:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:49.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:48:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:48:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:48:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:48:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:48:50.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2069 2024-09-16T12:48:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2068 2024-09-16T12:48:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2066 2024-09-16T12:48:51.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2067 2024-09-16T12:48:51.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:51.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:48:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:48:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:48:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:48:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:48:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:48:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2076 2024-09-16T12:48:51.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2075 2024-09-16T12:48:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:48:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:48:51.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:48:51.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:51.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:48:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:48:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:48:53.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2079 2024-09-16T12:48:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2080 2024-09-16T12:48:54.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:54.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2081 2024-09-16T12:48:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:48:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:48:54.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:48:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:48:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:48:54.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:48:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:48:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:48:54.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:48:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:54.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:48:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:48:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:48:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:48:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:48:54.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:48:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:48:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:48:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:48:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2085 2024-09-16T12:48:55.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:56.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2087 2024-09-16T12:48:56.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2090 2024-09-16T12:48:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2089 2024-09-16T12:48:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2088 2024-09-16T12:48:56.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:56.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:56.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:48:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:48:56.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:48:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2094 2024-09-16T12:48:57.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:57.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:48:57.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:57.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:57.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:48:57.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:48:57.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2095 2024-09-16T12:48:57.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:57.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:48:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:57.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:48:57.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:48:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:48:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:48:58.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:48:59.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2099 2024-09-16T12:48:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2100 2024-09-16T12:48:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:48:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:48:59.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:48:59.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2101 2024-09-16T12:48:59.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:59.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:48:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:48:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:48:59.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:48:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:48:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:48:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:48:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:48:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:48:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:48:59.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:48:59.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:48:59.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:49:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:49:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:49:00.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:49:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:49:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:49:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:49:00.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2105 2024-09-16T12:49:00.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:01.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2108 2024-09-16T12:49:01.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2106 2024-09-16T12:49:01.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2110 2024-09-16T12:49:01.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:01.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2109 2024-09-16T12:49:01.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:01.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:02.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2111 2024-09-16T12:49:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:02.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:49:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:49:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:49:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:49:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:49:02.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:49:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2112 2024-09-16T12:49:02.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:02.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:49:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:49:02.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:49:02.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:03.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:03.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:49:03.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:03.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:03.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:49:03.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:49:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2121 2024-09-16T12:49:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2119 2024-09-16T12:49:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2120 2024-09-16T12:49:03.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:03.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:49:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:04.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:49:04.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:49:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:49:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:49:04.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:49:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:49:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:49:04.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:49:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:49:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:49:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:49:04.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2125 2024-09-16T12:49:04.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:49:05.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:49:05.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:49:05.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:49:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:49:05.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:49:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2127 2024-09-16T12:49:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2130 2024-09-16T12:49:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2129 2024-09-16T12:49:05.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2128 2024-09-16T12:49:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:49:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:49:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:49:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2135 2024-09-16T12:49:06.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2136 2024-09-16T12:49:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:49:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:49:06.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:49:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:49:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:49:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:49:06.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:06.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:07.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:49:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:49:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:49:08.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:49:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:49:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:49:08.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2141 2024-09-16T12:49:08.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2140 2024-09-16T12:49:08.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2139 2024-09-16T12:49:08.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:49:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:49:08.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:49:08.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:09.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:49:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:49:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:49:09.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:49:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:49:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:49:09.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:09.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:49:09.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:09.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:09.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:49:09.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:49:09.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:49:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:49:09.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:49:09.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2143 2024-09-16T12:49:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:10.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2148 2024-09-16T12:49:11.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2150 2024-09-16T12:49:11.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:11.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2149 2024-09-16T12:49:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2146 2024-09-16T12:49:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:11.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:49:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:49:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:49:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2151 2024-09-16T12:49:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2155 2024-09-16T12:49:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:49:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:49:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:49:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:49:12.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:12.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:12.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:49:12.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:49:12.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:12.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:49:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:49:12.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:49:13.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2159 2024-09-16T12:49:14.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2160 2024-09-16T12:49:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:49:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:49:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:49:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:14.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2161 2024-09-16T12:49:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:49:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:49:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:49:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:14.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:49:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:49:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:49:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:49:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:49:14.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:49:15.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2165 2024-09-16T12:49:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:49:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:49:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:49:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:15.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:49:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:49:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:49:15.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2167 2024-09-16T12:49:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2168 2024-09-16T12:49:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2169 2024-09-16T12:49:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2170 2024-09-16T12:49:16.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:16.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:16.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2176 2024-09-16T12:49:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2174 2024-09-16T12:49:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:49:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:49:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:49:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:49:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:49:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:49:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:17.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:49:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:49:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:49:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:49:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:49:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:49:19.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2179 2024-09-16T12:49:19.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2181 2024-09-16T12:49:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:49:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:49:19.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:49:19.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2180 2024-09-16T12:49:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:49:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:49:19.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:49:19.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:49:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:49:19.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:49:19.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:49:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:49:19.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:49:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:49:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:49:20.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:49:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:49:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:49:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:49:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2185 2024-09-16T12:49:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:21.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2187 2024-09-16T12:49:21.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2188 2024-09-16T12:49:21.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2190 2024-09-16T12:49:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:21.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2189 2024-09-16T12:49:21.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:22.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2191 2024-09-16T12:49:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:49:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:49:22.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:49:22.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2192 2024-09-16T12:49:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:49:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:49:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:49:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:49:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:49:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:49:22.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:22.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:49:23.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:49:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:49:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2200 2024-09-16T12:49:23.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2201 2024-09-16T12:49:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:23.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2199 2024-09-16T12:49:23.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:49:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:49:24.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:49:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:49:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:49:24.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:49:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:49:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:49:24.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:49:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:49:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:49:24.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:49:24.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2205 2024-09-16T12:49:24.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:49:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:49:25.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:49:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:49:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:49:25.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:49:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2209 2024-09-16T12:49:25.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2207 2024-09-16T12:49:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2208 2024-09-16T12:49:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2210 2024-09-16T12:49:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:49:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:49:26.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:49:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:49:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:49:26.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:49:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2215 2024-09-16T12:49:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2216 2024-09-16T12:49:26.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:26.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:49:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:49:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:49:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:49:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:49:27.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:49:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:49:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:49:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:49:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:49:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:49:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:49:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:49:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:49:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:49:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2219 2024-09-16T12:49:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2220 2024-09-16T12:49:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:49:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:49:28.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:49:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:29.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:29.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2221 2024-09-16T12:49:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2225 2024-09-16T12:49:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:49:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:49:29.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:49:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:49:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:49:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:49:30.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:31.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2228 2024-09-16T12:49:31.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2230 2024-09-16T12:49:31.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2226 2024-09-16T12:49:31.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2227 2024-09-16T12:49:31.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:31.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:31.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2235 2024-09-16T12:49:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2234 2024-09-16T12:49:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:49:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:32.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:49:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:49:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:49:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:49:32.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:49:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:49:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:49:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:49:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:49:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:49:33.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:49:33.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2241 2024-09-16T12:49:34.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2240 2024-09-16T12:49:34.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:34.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:34.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:49:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:49:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:49:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:49:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:49:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:49:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:49:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:34.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:34.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:49:34.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:49:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:49:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:49:34.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:49:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2239 2024-09-16T12:49:34.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2245 2024-09-16T12:49:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:49:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:49:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:49:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:49:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:35.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:49:35.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:49:36.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2250 2024-09-16T12:49:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2246 2024-09-16T12:49:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2247 2024-09-16T12:49:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2248 2024-09-16T12:49:36.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:36.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:36.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:36.547 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:49:36.535+0000 7f61b1855700 -1 reset not still connected to 0x557b440d8c30 2024-09-16T12:49:36.547 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:49:36.535+0000 7f61b1855700 -1 reset not still connected to 0x557b44ed84e0 2024-09-16T12:49:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:49:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:49:37.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:49:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:49:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:37.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:49:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:49:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2255 2024-09-16T12:49:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2256 2024-09-16T12:49:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:37.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:49:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:49:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:49:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:49:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:49:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:49:39.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2261 2024-09-16T12:49:39.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2260 2024-09-16T12:49:39.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2259 2024-09-16T12:49:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:49:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:39.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:49:39.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:49:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:49:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:39.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:49:39.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:49:39.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:39.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:49:39.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:39.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:39.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:49:39.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:49:39.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:39.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:49:39.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:49:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:49:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:49:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:49:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:49:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:49:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:49:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:49:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2265 2024-09-16T12:49:40.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:41.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2267 2024-09-16T12:49:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2268 2024-09-16T12:49:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2269 2024-09-16T12:49:41.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:41.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2270 2024-09-16T12:49:41.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:41.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:49:42.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:49:42.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:49:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:42.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:49:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:49:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:49:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:49:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:49:42.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:49:42.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2275 2024-09-16T12:49:42.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2276 2024-09-16T12:49:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:49:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:49:43.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:49:43.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2280 2024-09-16T12:49:43.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2279 2024-09-16T12:49:43.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2281 2024-09-16T12:49:43.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:43.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:44.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:49:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:49:44.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:49:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:44.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:49:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:49:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:49:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:49:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:49:44.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:49:44.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:49:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:49:44.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:49:44.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2285 2024-09-16T12:49:44.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:49:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:49:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:49:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:49:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:49:45.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:49:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2288 2024-09-16T12:49:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2290 2024-09-16T12:49:45.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2287 2024-09-16T12:49:45.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:45.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2289 2024-09-16T12:49:46.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:46.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:46.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:49:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:49:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:49:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:49:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:49:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:49:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2296 2024-09-16T12:49:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:49:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:49:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:49:46.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2295 2024-09-16T12:49:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:49:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:49:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:49:48.618 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:49:48.607+0000 7f61b1855700 -1 reset not still connected to 0x557b440d8c30 2024-09-16T12:49:48.618 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:49:48.607+0000 7f61b1855700 -1 reset not still connected to 0x557b44ed84e0 2024-09-16T12:49:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2299 2024-09-16T12:49:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:49:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:49:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:49:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2300 2024-09-16T12:49:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:49:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:49:49.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:49:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2301 2024-09-16T12:49:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:49.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:49:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:49:49.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:49:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:49:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:49:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:49:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:49:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:49:49.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:49:50.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2305 2024-09-16T12:49:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:49:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:49:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:49:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:51.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2309 2024-09-16T12:49:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2307 2024-09-16T12:49:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2308 2024-09-16T12:49:51.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2310 2024-09-16T12:49:51.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:51.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:51.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:49:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:49:52.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:49:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2316 2024-09-16T12:49:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:52.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:49:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:49:52.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:49:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:49:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:49:52.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:49:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2315 2024-09-16T12:49:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:52.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:53.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:49:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:49:53.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:49:54.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2321 2024-09-16T12:49:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:49:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:49:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:49:54.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2319 2024-09-16T12:49:54.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2320 2024-09-16T12:49:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:49:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:49:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:49:54.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:54.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:49:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:49:54.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:49:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:49:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:49:54.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:49:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2325 2024-09-16T12:49:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:49:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:49:55.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:49:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:49:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:49:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:49:56.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2330 2024-09-16T12:49:56.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2327 2024-09-16T12:49:56.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2328 2024-09-16T12:49:56.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2329 2024-09-16T12:49:56.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:56.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:56.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:57.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2335 2024-09-16T12:49:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2336 2024-09-16T12:49:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:49:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:49:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:49:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:57.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:57.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:49:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:49:57.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:49:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:49:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:49:57.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:49:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:49:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:49:58.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:49:59.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2340 2024-09-16T12:49:59.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2341 2024-09-16T12:49:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2339 2024-09-16T12:49:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:49:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:49:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:49:59.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:49:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:49:59.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:49:59.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:49:59.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:59.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:49:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:49:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:49:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:49:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:49:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:49:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:49:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:49:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:50:00.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:50:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:50:00.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:50:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:50:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:50:00.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:50:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2345 2024-09-16T12:50:00.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:01.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2347 2024-09-16T12:50:01.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2350 2024-09-16T12:50:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:01.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:01.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2349 2024-09-16T12:50:01.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2348 2024-09-16T12:50:01.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:01.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:02.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2352 2024-09-16T12:50:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:50:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:50:02.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:50:02.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2353 2024-09-16T12:50:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:50:02.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:50:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:50:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:02.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:50:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:50:02.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:50:02.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:02.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:50:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:50:03.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:50:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2360 2024-09-16T12:50:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2359 2024-09-16T12:50:03.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2361 2024-09-16T12:50:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:50:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:04.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:50:04.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:50:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:50:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:50:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:50:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:50:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:50:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:50:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:50:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:50:04.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:50:04.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2365 2024-09-16T12:50:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:05.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:50:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:50:05.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:50:05.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:50:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:50:05.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:50:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2367 2024-09-16T12:50:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2370 2024-09-16T12:50:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2368 2024-09-16T12:50:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2369 2024-09-16T12:50:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:06.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:06.523 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:50:06.514+0000 7f74def64700 -1 reset not still connected to 0x5594601c80d0 2024-09-16T12:50:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2375 2024-09-16T12:50:06.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2376 2024-09-16T12:50:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:50:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:50:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:50:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:50:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:50:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:50:07.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:07.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:50:07.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:50:07.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:50:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:50:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:50:07.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:50:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2380 2024-09-16T12:50:08.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2381 2024-09-16T12:50:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2379 2024-09-16T12:50:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:50:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:50:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:50:09.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:50:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:09.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:50:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:50:09.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:50:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:50:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:50:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:50:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:50:09.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:50:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:09.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:50:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:50:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:50:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:50:10.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:50:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:50:10.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2385 2024-09-16T12:50:10.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:10.029 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:50:10.019+0000 7f61b1855700 -1 reset not still connected to 0x557b44ed84e0 2024-09-16T12:50:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2388 2024-09-16T12:50:11.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2390 2024-09-16T12:50:11.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:11.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2389 2024-09-16T12:50:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2387 2024-09-16T12:50:11.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:12.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:50:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:50:12.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:50:12.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2396 2024-09-16T12:50:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:50:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:50:12.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:50:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2395 2024-09-16T12:50:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:50:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:12.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:50:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:50:12.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:12.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:50:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:50:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:50:14.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2400 2024-09-16T12:50:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:50:14.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:50:14.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:50:14.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2399 2024-09-16T12:50:14.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:50:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:50:14.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:50:14.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:14.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2401 2024-09-16T12:50:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:50:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:50:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:50:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:50:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:50:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:50:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:50:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:50:15.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:50:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:50:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:15.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:50:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:50:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2405 2024-09-16T12:50:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:16.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2407 2024-09-16T12:50:16.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2410 2024-09-16T12:50:16.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2408 2024-09-16T12:50:16.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2409 2024-09-16T12:50:16.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:16.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:16.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:17.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2416 2024-09-16T12:50:17.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:50:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:50:17.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:50:17.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2415 2024-09-16T12:50:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:50:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:50:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:50:17.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:17.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:50:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:50:17.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:50:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:18.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:50:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:50:18.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:50:19.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2419 2024-09-16T12:50:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:50:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:50:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:50:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:50:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:50:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:50:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2421 2024-09-16T12:50:19.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2420 2024-09-16T12:50:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:50:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:19.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:50:19.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:50:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:50:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:50:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:50:19.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:19.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:19.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2425 2024-09-16T12:50:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:50:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:50:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:50:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:50:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:50:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:50:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:21.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2429 2024-09-16T12:50:21.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2427 2024-09-16T12:50:21.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2428 2024-09-16T12:50:21.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2430 2024-09-16T12:50:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:21.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:50:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:22.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:50:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:50:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:50:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:50:22.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:50:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2435 2024-09-16T12:50:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2436 2024-09-16T12:50:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:50:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:50:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:50:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:22.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:23.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:50:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:50:23.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:50:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2441 2024-09-16T12:50:23.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2439 2024-09-16T12:50:23.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2440 2024-09-16T12:50:23.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:23.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:23.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:50:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:50:24.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:50:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:50:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:50:24.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:50:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:50:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:50:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:50:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:50:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:24.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:50:24.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:50:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2445 2024-09-16T12:50:24.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:25.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:50:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:50:25.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:50:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:50:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:25.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:50:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:50:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2450 2024-09-16T12:50:26.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2448 2024-09-16T12:50:26.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2449 2024-09-16T12:50:26.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2447 2024-09-16T12:50:26.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:26.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:26.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:26.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:50:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:50:26.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:50:26.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:50:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:50:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:50:26.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:26.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:50:26.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:26.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:26.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:50:26.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:50:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2455 2024-09-16T12:50:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2456 2024-09-16T12:50:27.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:27.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:50:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:50:27.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:50:29.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2461 2024-09-16T12:50:29.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2460 2024-09-16T12:50:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2459 2024-09-16T12:50:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:50:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:50:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:50:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:50:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:50:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:50:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:50:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:50:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:50:29.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:50:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:50:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:50:29.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:30.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:50:30.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2465 2024-09-16T12:50:30.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:31.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2470 2024-09-16T12:50:31.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2469 2024-09-16T12:50:31.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2467 2024-09-16T12:50:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:31.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2468 2024-09-16T12:50:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:31.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:31.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2475 2024-09-16T12:50:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2476 2024-09-16T12:50:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:50:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:50:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:50:32.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:50:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:50:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:50:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:50:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:50:32.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:50:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:50:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:50:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:50:34.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2480 2024-09-16T12:50:34.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2479 2024-09-16T12:50:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2481 2024-09-16T12:50:34.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:50:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:50:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:50:34.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:50:34.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:50:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:50:34.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:50:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:50:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:50:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:50:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:50:34.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:50:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:35.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:50:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:50:35.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:50:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:50:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:50:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:50:35.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2485 2024-09-16T12:50:35.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:35.587 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:50:35.573+0000 7f74def64700 -1 reset not still connected to 0x559455bdc680 2024-09-16T12:50:35.587 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:50:35.573+0000 7f74def64700 -1 reset not still connected to 0x559457985380 2024-09-16T12:50:35.587 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:50:35.573+0000 7f74def64700 -1 reset not still connected to 0x55945f331790 2024-09-16T12:50:36.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2488 2024-09-16T12:50:36.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2489 2024-09-16T12:50:36.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:36.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2487 2024-09-16T12:50:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2490 2024-09-16T12:50:36.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:36.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:36.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:50:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:50:37.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:50:37.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2495 2024-09-16T12:50:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2496 2024-09-16T12:50:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:50:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:50:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:50:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:37.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:50:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:50:37.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:50:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:37.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:38.065 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:50:38.058+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0ea0 2024-09-16T12:50:38.065 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:50:38.058+0000 7f61b1855700 -1 reset not still connected to 0x557b22af11e0 2024-09-16T12:50:38.066 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:50:38.058+0000 7f61b1855700 -1 reset not still connected to 0x557b39993110 2024-09-16T12:50:38.069 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:50:38.058+0000 7f61b1855700 -1 reset not still connected to 0x557b440d8c30 2024-09-16T12:50:38.070 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:50:38.058+0000 7f61b1855700 -1 reset not still connected to 0x557b44ed84e0 2024-09-16T12:50:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:50:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:50:38.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:50:39.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2500 2024-09-16T12:50:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2499 2024-09-16T12:50:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2501 2024-09-16T12:50:39.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:50:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:50:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:50:39.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:39.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:50:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:50:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:50:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:50:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:50:39.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:50:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:50:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:50:39.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:50:40.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2504 2024-09-16T12:50:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:50:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:50:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:50:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:50:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:50:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:50:40.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:41.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2509 2024-09-16T12:50:41.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2508 2024-09-16T12:50:41.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2510 2024-09-16T12:50:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:41.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2507 2024-09-16T12:50:41.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:41.593 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:50:41.581+0000 7f74def64700 -1 reset not still connected to 0x559455bdc680 2024-09-16T12:50:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2515 2024-09-16T12:50:42.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:50:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:50:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:50:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2516 2024-09-16T12:50:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:50:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:42.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:50:42.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:50:42.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:50:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:50:42.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:50:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:50:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:43.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:50:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:50:43.547 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:50:43.538+0000 7f4071c2f700 -1 reset not still connected to 0x5635b7f441a0 2024-09-16T12:50:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2519 2024-09-16T12:50:43.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2520 2024-09-16T12:50:43.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2521 2024-09-16T12:50:44.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:44.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:44.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:50:44.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:50:44.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:50:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:50:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:50:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:50:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:50:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:50:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:50:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:50:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:50:44.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:50:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2525 2024-09-16T12:50:44.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:45.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:50:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:50:45.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:50:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:50:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:50:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:50:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2530 2024-09-16T12:50:45.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2528 2024-09-16T12:50:46.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2527 2024-09-16T12:50:46.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2529 2024-09-16T12:50:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:46.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:46.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:46.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2535 2024-09-16T12:50:47.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2536 2024-09-16T12:50:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:50:47.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:47.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:47.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:50:47.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:50:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:50:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:50:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:50:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:50:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:50:47.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:50:47.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:47.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:50:47.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:47.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:47.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:50:47.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:50:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:50:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:50:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:50:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2539 2024-09-16T12:50:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2540 2024-09-16T12:50:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:50:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:50:49.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:50:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:50:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:50:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:50:49.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:49.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:50:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:50:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:50:49.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2541 2024-09-16T12:50:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2545 2024-09-16T12:50:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:50:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:50:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:50:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:50:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:50:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:50:50.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:51.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2550 2024-09-16T12:50:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2548 2024-09-16T12:50:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2547 2024-09-16T12:50:51.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:51.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:51.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2549 2024-09-16T12:50:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:52.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:50:52.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 62890 2024-09-16T12:50:52.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2556 2024-09-16T12:50:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:50:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:50:52.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:50:52.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2554 2024-09-16T12:50:52.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:50:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:50:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:50:52.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:50:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:50:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:50:54.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2559 2024-09-16T12:50:54.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2560 2024-09-16T12:50:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:50:54.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 63036 2024-09-16T12:50:54.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:50:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:50:54.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:50:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:50:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:54.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:50:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:50:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:50:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:50:54.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:50:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2563 2024-09-16T12:50:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:50:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:50:55.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:50:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:50:55.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 63190 2024-09-16T12:50:55.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:56.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2565 2024-09-16T12:50:56.071 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:50:56.062+0000 7f4071c2f700 -1 reset not still connected to 0x5635a0aa0ea0 2024-09-16T12:50:56.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2567 2024-09-16T12:50:56.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2566 2024-09-16T12:50:56.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:56.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2571 2024-09-16T12:50:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:50:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:50:57.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:50:57.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:57.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:50:57.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 63345 2024-09-16T12:50:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:50:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:50:58.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:50:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2573 2024-09-16T12:50:59.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:50:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:50:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 63500 2024-09-16T12:50:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:50:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:50:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:50:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:50:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:50:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:50:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:50:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:50:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:51:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:51:00.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 63675 2024-09-16T12:51:00.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2575 2024-09-16T12:51:00.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:51:01.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2578 2024-09-16T12:51:01.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2577 2024-09-16T12:51:01.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:51:01.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:51:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:51:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:51:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:51:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:51:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:51:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:51:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:51:03.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 63891 2024-09-16T12:51:03.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2581 2024-09-16T12:51:03.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:51:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:51:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:51:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:51:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:51:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:51:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:51:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:51:04.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 64069 2024-09-16T12:51:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2583 2024-09-16T12:51:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:51:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:51:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 64367 2024-09-16T12:51:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T12:51:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:51:08.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 62757 2024-09-16T12:53:24.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:53:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:53:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:53:24.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 62891 2024-09-16T12:53:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:53:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:53:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:53:25.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 63037 2024-09-16T12:53:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:53:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:53:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:53:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 63191 2024-09-16T12:53:28.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:53:28.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:53:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:53:28.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 63346 2024-09-16T12:53:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:53:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:53:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:53:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 63501 2024-09-16T12:53:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:53:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:53:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:53:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 63676 2024-09-16T12:53:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:53:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:53:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:53:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 63892 2024-09-16T12:53:36.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:53:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:53:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:53:36.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 64070 2024-09-16T12:53:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:53:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:53:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:53:38.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 64368 2024-09-16T12:53:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T12:53:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T12:53:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T12:53:40.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:53:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:53:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:53:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 1 2024-09-16T12:53:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd0 2024-09-16T12:53:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:53:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary1 2024-09-16T12:53:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary1 2024-09-16T12:53:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:53:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary1 2024-09-16T12:53:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:53:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 2 2024-09-16T12:53:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:53:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd1 2024-09-16T12:53:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary2 2024-09-16T12:53:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary2 2024-09-16T12:53:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:53:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary2 2024-09-16T12:53:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:53:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 3 2024-09-16T12:53:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd2 2024-09-16T12:53:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary3 2024-09-16T12:53:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary3 2024-09-16T12:53:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:53:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary3 2024-09-16T12:53:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:53:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:53:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:53:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:53:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 5 2024-09-16T12:53:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd4 2024-09-16T12:53:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary5 2024-09-16T12:53:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary5 2024-09-16T12:53:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:53:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary5 2024-09-16T12:53:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:53:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:53:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 6 2024-09-16T12:53:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:53:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd5 2024-09-16T12:53:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary6 2024-09-16T12:53:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:53:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary6 2024-09-16T12:53:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:53:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 7 2024-09-16T12:53:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary6 2024-09-16T12:53:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:53:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd6 2024-09-16T12:53:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary7 2024-09-16T12:53:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary7 2024-09-16T12:53:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:53:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary7 2024-09-16T12:53:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:53:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 8 2024-09-16T12:53:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:53:40.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd7 2024-09-16T12:53:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:53:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary8 2024-09-16T12:53:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary8 2024-09-16T12:53:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:53:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary8 2024-09-16T12:53:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 9 2024-09-16T12:53:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T12:53:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:53:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 70216 2024-09-16T12:53:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 10 2024-09-16T12:53:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd9 2024-09-16T12:53:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 4 2024-09-16T12:53:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd8 2024-09-16T12:53:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary9 2024-09-16T12:53:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary9 2024-09-16T12:53:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:53:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary9 2024-09-16T12:53:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd3 2024-09-16T12:53:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary10 2024-09-16T12:53:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary10 2024-09-16T12:53:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:53:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary10 2024-09-16T12:53:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary4 2024-09-16T12:53:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary4 2024-09-16T12:53:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T12:53:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary4 2024-09-16T12:53:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:53:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd5 2024-09-16T12:53:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd1 2024-09-16T12:53:40.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:53:40.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd2 2024-09-16T12:53:40.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:53:40.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:53:40.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd7 2024-09-16T12:53:40.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd4 2024-09-16T12:53:40.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:53:40.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:53:40.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd3 2024-09-16T12:53:40.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd0 2024-09-16T12:53:40.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:53:40.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd8 2024-09-16T12:53:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:53:40.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd6 2024-09-16T12:53:40.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:53:40.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:53:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd9 2024-09-16T12:54:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=8ccd2b1d27f7c9d89bf84d7adb22c6af 2024-09-16T12:54:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary3 2024-09-16T12:54:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=e90baeb1de83703427b62ae5fd2ddab8 2024-09-16T12:54:37.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary7 2024-09-16T12:54:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=a8a29ac1763dc1bf0d7e751cac12b6c2 2024-09-16T12:54:37.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary2 2024-09-16T12:54:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=5d935b7e472014244095439c5d2e79b6 2024-09-16T12:54:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary4 2024-09-16T12:54:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=c0e960550a19042eb11a659270978bde 2024-09-16T12:54:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary5 2024-09-16T12:54:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=f7c5d2d5fc7a952da5632bdef4b4a1dd 2024-09-16T12:54:37.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary9 2024-09-16T12:54:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=8d8245fea0e9ec7342f3b3695d600ad9 2024-09-16T12:54:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary1 2024-09-16T12:54:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=8d5b742d7a269c511a8ae602d7950dec 2024-09-16T12:54:37.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary6 2024-09-16T12:54:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=0d82bcf3841c016be2df5a6e95c968f7 2024-09-16T12:54:37.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary10 2024-09-16T12:54:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-16T12:54:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:54:38.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-16T12:54:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-16T12:54:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:54:38.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-16T12:54:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-16T12:54:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:54:38.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-16T12:54:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-16T12:54:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:54:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-16T12:54:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-16T12:54:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:54:38.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-16T12:54:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=b92db8d54caf32ca485b93013713f6c0 2024-09-16T12:54:38.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary8 2024-09-16T12:54:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-16T12:54:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:54:38.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-16T12:54:38.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-16T12:54:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:54:38.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-16T12:54:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-16T12:54:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:54:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-16T12:54:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-16T12:54:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:54:38.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-16T12:54:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-16T12:54:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:54:38.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-16T12:54:39.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:54:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:54:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:54:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:54:39.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:54:39.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:54:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-16T12:54:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:54:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:39.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:54:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:54:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-16T12:54:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:54:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:54:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-16T12:54:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:54:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-16T12:54:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:54:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:54:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:39.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:54:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-16T12:54:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:54:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:54:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-16T12:54:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:54:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:39.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:54:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-16T12:54:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:54:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:54:40.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:54:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:54:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T12:54:40.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-16T12:54:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:54:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:40.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:54:40.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-16T12:54:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:54:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:40.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:54:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-16T12:54:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:54:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:40.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:54:40.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-16T12:54:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:54:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-16T12:54:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-16T12:54:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-16T12:54:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:40.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:54:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:54:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-16T12:54:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:40.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:54:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-16T12:54:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:54:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:54:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:40.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-16T12:54:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T12:54:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:54:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:40.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T12:54:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T12:54:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:40.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T12:54:40.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T12:54:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T12:54:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:40.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T12:54:41.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T12:54:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: afa1056f-626c-48ac-b10e-3d158ee2f57a 2024-09-16T12:54:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:54:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:54:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:38 2024-09-16T12:54:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:54:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":194560.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491059,"remote_snapshot_timestamp":1726491059,"replay_state":"idle"} 2024-09-16T12:54:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:23' 2024-09-16T12:54:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T12:54:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: dd6269fd-f848-44f5-b3e7-af6943c54ccb 2024-09-16T12:54:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:54:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:54:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:41.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:38 2024-09-16T12:54:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:54:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:40' 2024-09-16T12:54:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T12:54:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 4734d245-46e2-4286-95a3-3cc562d928fa 2024-09-16T12:54:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:54:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:54:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:41.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:38 2024-09-16T12:54:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:54:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2142208.0,"last_snapshot_bytes":2588672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491062,"remote_snapshot_timestamp":1726491062,"replay_state":"idle"} 2024-09-16T12:54:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:23' 2024-09-16T12:54:41.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T12:54:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 8d0e5cc3-e3f8-4977-9e72-a3c1492e4e85 2024-09-16T12:54:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:54:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:54:41.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:38 2024-09-16T12:54:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:54:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:40' 2024-09-16T12:54:41.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T12:54:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: fec07e41-ee3f-4c53-baf9-b248297f6077 2024-09-16T12:54:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:54:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:54:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:38 2024-09-16T12:54:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:54:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":962560.0,"last_snapshot_bytes":1744896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491050,"remote_snapshot_timestamp":1726491050,"replay_state":"idle"} 2024-09-16T12:54:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:23' 2024-09-16T12:54:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:41.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T12:54:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: c1088e42-e596-4d28-9923-42d6045d5571 2024-09-16T12:54:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:54:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:54:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:38 2024-09-16T12:54:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:54:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":995328.0,"last_snapshot_bytes":1789952,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491047,"remote_snapshot_timestamp":1726491047,"replay_state":"idle"} 2024-09-16T12:54:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:23' 2024-09-16T12:54:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T12:54:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: aedce2c3-b936-41c9-95f9-3b2be5e43d04 2024-09-16T12:54:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:54:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:54:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:38 2024-09-16T12:54:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:54:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:40' 2024-09-16T12:54:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:54:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:54:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:54:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:54:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:41.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:54:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:54:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:54:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:54:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:54:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:54:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:54:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:54:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:54:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:54:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-16T12:54:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:54:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:41.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T12:54:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-16T12:54:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:54:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T12:54:41.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-16T12:54:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:54:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T12:54:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T12:54:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: c82934ef-5cb6-459e-a80f-ec8e46fbc42a 2024-09-16T12:54:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:54:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:54:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:38 2024-09-16T12:54:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:54:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:41' 2024-09-16T12:54:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:54:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:41.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:54:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T12:54:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: e1674f59-5785-4885-8027-0915586882c9 2024-09-16T12:54:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:54:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:54:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:38 2024-09-16T12:54:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:54:42.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:41' 2024-09-16T12:54:42.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:54:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:42.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:54:42.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T12:54:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 729e14b6-bf4a-4677-9904-ec7f7484589d 2024-09-16T12:54:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:54:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:54:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:38 2024-09-16T12:54:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:54:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:41' 2024-09-16T12:54:42.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:54:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:42.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T12:54:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-16T12:54:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:54:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:43.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-16T12:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-16T12:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:43.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-16T12:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:43.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T12:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-16T12:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T12:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T12:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:43.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T12:54:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-16T12:54:43.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T12:54:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:54:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-16T12:54:43.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:54:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T12:54:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:43.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T12:54:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T12:54:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: afa1056f-626c-48ac-b10e-3d158ee2f57a 2024-09-16T12:54:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:54:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:54:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:38 2024-09-16T12:54:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:54:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:41' 2024-09-16T12:54:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:54:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 4 2024-09-16T12:54:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T12:54:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: fec07e41-ee3f-4c53-baf9-b248297f6077 2024-09-16T12:54:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:54:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:54:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:38 2024-09-16T12:54:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:54:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:41' 2024-09-16T12:54:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T12:54:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: aedce2c3-b936-41c9-95f9-3b2be5e43d04 2024-09-16T12:54:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:54:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:54:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:38 2024-09-16T12:54:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:54:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:40' 2024-09-16T12:54:43.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:54:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 4 2024-09-16T12:54:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:54:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 4 2024-09-16T12:54:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T12:54:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: c1088e42-e596-4d28-9923-42d6045d5571 2024-09-16T12:54:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:54:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:54:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:38 2024-09-16T12:54:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:54:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:41' 2024-09-16T12:54:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T12:54:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 4734d245-46e2-4286-95a3-3cc562d928fa 2024-09-16T12:54:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:54:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:54:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:38 2024-09-16T12:54:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:54:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:41' 2024-09-16T12:54:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T12:54:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 8d0e5cc3-e3f8-4977-9e72-a3c1492e4e85 2024-09-16T12:54:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:54:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:54:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:38 2024-09-16T12:54:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:54:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:40' 2024-09-16T12:54:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T12:54:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: dd6269fd-f848-44f5-b3e7-af6943c54ccb 2024-09-16T12:54:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:54:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:54:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:38 2024-09-16T12:54:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T12:54:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:40' 2024-09-16T12:54:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:54:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 4 2024-09-16T12:54:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:54:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 4 2024-09-16T12:54:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:54:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 4 2024-09-16T12:54:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T12:54:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:43.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 4 2024-09-16T12:54:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-16T12:54:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:54:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:43.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:43.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T12:54:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-16T12:54:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:54:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:44.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:44.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T12:54:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-16T12:54:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:54:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:44.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:44.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T12:54:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T12:54:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: c82934ef-5cb6-459e-a80f-ec8e46fbc42a 2024-09-16T12:54:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:44.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43 2024-09-16T12:54:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43' 2024-09-16T12:54:44.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:44.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:54:44.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:54:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:54:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:54:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:44.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:54:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:44.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:54:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:54:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:54:44.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:54:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T12:54:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: e1674f59-5785-4885-8027-0915586882c9 2024-09-16T12:54:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:44.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43 2024-09-16T12:54:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43' 2024-09-16T12:54:44.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:44.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:54:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T12:54:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 729e14b6-bf4a-4677-9904-ec7f7484589d 2024-09-16T12:54:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43 2024-09-16T12:54:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43' 2024-09-16T12:54:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:54:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:54:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:54:44.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:54:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:54:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:44.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:54:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:54:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:54:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:54:44.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:54:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:54:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:54:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:54:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:44.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:54:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:54:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:54:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:54:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-16 12:54:43' 2024-09-16T12:54:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-16 12:54:43' 2024-09-16T12:54:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:54:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:54:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:54:43' 2024-09-16T12:54:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-16 12:54:43' 2024-09-16T12:54:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:54:43' 2024-09-16T12:54:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:54:44.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:54:43' 2024-09-16T12:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:44.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:54:44.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary6e1674f59-5785-4885-8027-0915586882c9up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary94734d245-46e2-4286-95a3-3cc562d928faup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43' 2024-09-16T12:54:44.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:54:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary6e1674f59-5785-4885-8027-0915586882c9up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary94734d245-46e2-4286-95a3-3cc562d928faup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43' 2024-09-16T12:54:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2<+ local last_update state 2024-09-16T12:54:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T12:54:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mirror_uuids>c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary6e1674f59-5785-4885-8027-0915586882c9up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary94734d245-46e2-4286-95a3-3cc562d928faup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43' 2024-09-16T12:54:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:54:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T12:54:44.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T12:54:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T12:54:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T12:54:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T12:54:44.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:54:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:54:44.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-16T12:54:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:54:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:54:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:54:44.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:54:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:54:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:54:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:44.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:44.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-16T12:54:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:54:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:54:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-16T12:54:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:54:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:54:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-16T12:54:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:54:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:45.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-16T12:54:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-16T12:54:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:54:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:45.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-16T12:54:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:54:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:45.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-16T12:54:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:45.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-16T12:54:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T12:54:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: e1674f59-5785-4885-8027-0915586882c9 2024-09-16T12:54:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:45.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43 2024-09-16T12:54:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:54:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43' 2024-09-16T12:54:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:45.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T12:54:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 729e14b6-bf4a-4677-9904-ec7f7484589d 2024-09-16T12:54:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43 2024-09-16T12:54:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:54:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43' 2024-09-16T12:54:45.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:54:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:54:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T12:54:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: c82934ef-5cb6-459e-a80f-ec8e46fbc42a 2024-09-16T12:54:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:45.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43 2024-09-16T12:54:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:54:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43' 2024-09-16T12:54:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:45.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:54:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:54:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:54:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:54:45.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:54:45.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:54:45.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:45.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:54:45.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:54:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:54:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:54:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:45.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:54:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:54:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:54:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:54:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:54:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:54:45.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:54:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:54:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:54:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:54:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:54:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:54:45.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:54:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:54:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:54:45.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-16 12:54:43' 2024-09-16T12:54:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:54:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:54:43' 2024-09-16T12:54:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-16 12:54:43' 2024-09-16T12:54:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:54:45.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:54:43' 2024-09-16T12:54:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:54:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:54:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-16 12:54:43' 2024-09-16T12:54:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:54:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:45.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:54:43' 2024-09-16T12:54:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:45.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:54:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary6e1674f59-5785-4885-8027-0915586882c9up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary94734d245-46e2-4286-95a3-3cc562d928faup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43' 2024-09-16T12:54:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:54:45.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T12:54:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:45.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T12:54:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary6e1674f59-5785-4885-8027-0915586882c9up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary94734d245-46e2-4286-95a3-3cc562d928faup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43' 2024-09-16T12:54:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:54:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:54:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T12:54:45.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:54:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary6e1674f59-5785-4885-8027-0915586882c9up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43image-primary94734d245-46e2-4286-95a3-3cc562d928faup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:43' 2024-09-16T12:54:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:54:45.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T12:54:45.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:45.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:45.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-16T12:54:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:54:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:54:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-16T12:54:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T12:54:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:45.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T12:54:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:54:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:54:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-16T12:54:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:54:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:54:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-16T12:54:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:54:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:45.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:54:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-16T12:54:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:54:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:54:45.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-16T12:54:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:54:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:54:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:54:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:54:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:54:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary10 2024-09-16T12:54:46.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary6 2024-09-16T12:54:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:54:46.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary8 2024-09-16T12:54:46.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd0 2024-09-16T12:54:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:54:46.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd0 2024-09-16T12:54:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd1 2024-09-16T12:54:46.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd1 2024-09-16T12:54:46.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:54:46.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd2 2024-09-16T12:54:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:54:46.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd2 2024-09-16T12:54:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-16T12:54:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:47.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:54:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T12:54:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-16T12:54:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:54:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:47.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-16T12:54:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T12:54:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:54:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T12:54:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-16T12:54:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:54:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:47.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-16T12:54:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:47.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-16T12:54:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:54:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:54:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:47.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-16T12:54:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:54:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T12:54:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:47.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T12:54:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T12:54:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:47.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T12:54:47.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T12:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: aedce2c3-b936-41c9-95f9-3b2be5e43d04 2024-09-16T12:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43 2024-09-16T12:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43' 2024-09-16T12:54:47.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:54:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:54:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:54:47.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:54:47.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:54:47.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:47.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:54:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:54:47.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:54:47.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:54:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-16 12:54:43' 2024-09-16T12:54:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:54:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:54:43' 2024-09-16T12:54:47.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:54:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T12:54:47.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 8d0e5cc3-e3f8-4977-9e72-a3c1492e4e85 2024-09-16T12:54:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43 2024-09-16T12:54:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43' 2024-09-16T12:54:47.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T12:54:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 4734d245-46e2-4286-95a3-3cc562d928fa 2024-09-16T12:54:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43 2024-09-16T12:54:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43' 2024-09-16T12:54:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:47.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:54:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:54:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:54:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T12:54:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: fec07e41-ee3f-4c53-baf9-b248297f6077 2024-09-16T12:54:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43 2024-09-16T12:54:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43' 2024-09-16T12:54:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:54:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T12:54:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: afa1056f-626c-48ac-b10e-3d158ee2f57a 2024-09-16T12:54:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43 2024-09-16T12:54:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43' 2024-09-16T12:54:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:54:47.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:54:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:47.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:54:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T12:54:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: c1088e42-e596-4d28-9923-42d6045d5571 2024-09-16T12:54:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43 2024-09-16T12:54:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:47.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43' 2024-09-16T12:54:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:54:47.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:54:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:54:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:54:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:54:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:54:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:54:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:54:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:54:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T12:54:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: dd6269fd-f848-44f5-b3e7-af6943c54ccb 2024-09-16T12:54:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43 2024-09-16T12:54:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:47.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:54:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:43' 2024-09-16T12:54:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:47.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-16 12:54:43' 2024-09-16T12:54:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:54:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:54:43' 2024-09-16T12:54:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:54:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:54:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:54:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:54:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:54:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:54:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:54:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:47.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:54:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:54:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:54:47.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:54:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:54:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:54:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-16 12:54:43' 2024-09-16T12:54:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:54:47.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:54:43' 2024-09-16T12:54:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:54:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:54:47.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:54:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:54:47.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:54:47.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:54:47.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:54:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:54:47.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:54:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:54:47.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:54:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:54:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:54:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:54:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:47.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:54:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:54:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:47.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-16 12:54:43' 2024-09-16T12:54:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:54:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:54:43' 2024-09-16T12:54:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:54:47.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:47.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:54:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-16 12:54:43' 2024-09-16T12:54:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:54:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:54:43' 2024-09-16T12:54:47.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:54:47.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:54:47.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:54:47.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:54:47.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:54:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:47.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:54:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:54:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:54:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:54:47.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:54:47.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:54:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-16 12:54:43' 2024-09-16T12:54:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:54:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:54:43' 2024-09-16T12:54:47.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-16 12:54:43' 2024-09-16T12:54:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:54:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:47.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:54:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:54:43' 2024-09-16T12:54:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:47.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:54:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:47.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:54:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary6e1674f59-5785-4885-8027-0915586882c9up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary94734d245-46e2-4286-95a3-3cc562d928faup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43' 2024-09-16T12:54:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:54:47.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T12:54:47.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary6e1674f59-5785-4885-8027-0915586882c9up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary94734d245-46e2-4286-95a3-3cc562d928faup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43' 2024-09-16T12:54:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:54:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T12:54:47.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T12:54:47.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:54:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:47.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:54:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T12:54:47.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-16T12:54:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:54:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:47.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:47.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:54:47.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:54:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:47.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:54:47.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-16T12:54:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:54:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:47.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:47.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:54:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary6e1674f59-5785-4885-8027-0915586882c9up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary94734d245-46e2-4286-95a3-3cc562d928faup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43' 2024-09-16T12:54:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:54:47.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T12:54:47.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary6e1674f59-5785-4885-8027-0915586882c9up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary94734d245-46e2-4286-95a3-3cc562d928faup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43' 2024-09-16T12:54:47.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:54:47.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:47.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T12:54:47.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T12:54:47.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary6e1674f59-5785-4885-8027-0915586882c9up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary94734d245-46e2-4286-95a3-3cc562d928faup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43' 2024-09-16T12:54:47.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:54:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T12:54:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:54:47.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:47.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:54:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T12:54:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:47.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-16T12:54:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:54:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:47.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:54:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:47.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:54:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T12:54:47.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:54:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-16T12:54:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:54:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:47.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:54:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:54:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:47.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:54:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-16T12:54:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:54:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:54:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary6e1674f59-5785-4885-8027-0915586882c9up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary94734d245-46e2-4286-95a3-3cc562d928faup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43' 2024-09-16T12:54:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:54:47.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T12:54:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:47.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T12:54:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:54:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:47.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:54:47.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:47.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-16T12:54:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:54:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:54:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary6e1674f59-5785-4885-8027-0915586882c9up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43image-primary94734d245-46e2-4286-95a3-3cc562d928faup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 12:54:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 12:54:43' 2024-09-16T12:54:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:54:48.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T12:54:48.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:43' 2024-09-16T12:54:48.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T12:54:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:54:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:48.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:54:48.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-16T12:54:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:54:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:54:48.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:54:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-16T12:54:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:54:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-16T12:54:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-16T12:54:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:54:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-16T12:54:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-16T12:54:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:54:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:48.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-16T12:54:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-16T12:54:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:54:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-16T12:54:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:54:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:48.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-16T12:54:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:48.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-16T12:54:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T12:54:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: aedce2c3-b936-41c9-95f9-3b2be5e43d04 2024-09-16T12:54:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:48 2024-09-16T12:54:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:54:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:48' 2024-09-16T12:54:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:54:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:54:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:54:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:54:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:54:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:54:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:54:48.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:54:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:54:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:48' 2024-09-16T12:54:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-16 12:54:48' 2024-09-16T12:54:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:54:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:54:48' 2024-09-16T12:54:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:54:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-16T12:54:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:54:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:48.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:48.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:48.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-16T12:54:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T12:54:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: fec07e41-ee3f-4c53-baf9-b248297f6077 2024-09-16T12:54:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:48 2024-09-16T12:54:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:54:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:48' 2024-09-16T12:54:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T12:54:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: c1088e42-e596-4d28-9923-42d6045d5571 2024-09-16T12:54:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:48 2024-09-16T12:54:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:54:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:48' 2024-09-16T12:54:48.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:54:48.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:54:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:54:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:54:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T12:54:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: dd6269fd-f848-44f5-b3e7-af6943c54ccb 2024-09-16T12:54:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:48 2024-09-16T12:54:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:54:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:48' 2024-09-16T12:54:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:54:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:54:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:54:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:54:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:54:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:54:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:54:48.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:54:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:54:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:54:48.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:54:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:48.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:54:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T12:54:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 8d0e5cc3-e3f8-4977-9e72-a3c1492e4e85 2024-09-16T12:54:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:48 2024-09-16T12:54:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:54:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:48.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:48' 2024-09-16T12:54:48.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:48.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:48.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:54:48.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:54:48.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:54:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:54:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:54:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:48.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:48.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:54:48.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:54:48.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:54:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:54:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:54:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:54:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:48' 2024-09-16T12:54:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-16 12:54:48' 2024-09-16T12:54:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:54:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:54:48' 2024-09-16T12:54:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:54:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:54:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:48' 2024-09-16T12:54:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-16 12:54:48' 2024-09-16T12:54:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:54:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:54:48' 2024-09-16T12:54:48.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:54:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:54:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:54:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:54:48.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:54:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:48.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:54:48.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:54:48.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:54:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:48' 2024-09-16T12:54:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-16 12:54:48' 2024-09-16T12:54:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:54:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:54:48' 2024-09-16T12:54:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:54:48.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:48.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:54:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:48' 2024-09-16T12:54:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-16 12:54:48' 2024-09-16T12:54:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:54:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:54:48' 2024-09-16T12:54:48.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:54:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T12:54:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: afa1056f-626c-48ac-b10e-3d158ee2f57a 2024-09-16T12:54:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:48 2024-09-16T12:54:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:54:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:48' 2024-09-16T12:54:48.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:54:49.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:54:49.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:54:49.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:54:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:54:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:54:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:54:49.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:54:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:54:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:48' 2024-09-16T12:54:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-16 12:54:48' 2024-09-16T12:54:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:54:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:54:48' 2024-09-16T12:54:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:49.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:49.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:54:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING7342654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary6e1674f59-5785-4885-8027-0915586882c9up+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary94734d245-46e2-4286-95a3-3cc562d928faup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48' 2024-09-16T12:54:49.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:54:49.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T12:54:49.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:48' 2024-09-16T12:54:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T12:54:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:54:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:54:49.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:49.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-16T12:54:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:54:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:54:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-16T12:54:49.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-16T12:54:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:54:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:54:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:54:49.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:49.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:49.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-16T12:54:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING7342654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary6e1674f59-5785-4885-8027-0915586882c9up+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary94734d245-46e2-4286-95a3-3cc562d928faup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48' 2024-09-16T12:54:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:54:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING7342654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary6e1674f59-5785-4885-8027-0915586882c9up+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary94734d245-46e2-4286-95a3-3cc562d928faup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48' 2024-09-16T12:54:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:54:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T12:54:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T12:54:49.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:48' 2024-09-16T12:54:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:48' 2024-09-16T12:54:49.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T12:54:49.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T12:54:49.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:54:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:54:49.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:54:49.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:54:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-16T12:54:49.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:54:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:54:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-16T12:54:49.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:49.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-16T12:54:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:54:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:54:49.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-16T12:54:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING7342654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary6e1674f59-5785-4885-8027-0915586882c9up+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary94734d245-46e2-4286-95a3-3cc562d928faup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48' 2024-09-16T12:54:49.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:54:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING7342654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary6e1674f59-5785-4885-8027-0915586882c9up+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary94734d245-46e2-4286-95a3-3cc562d928faup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48' 2024-09-16T12:54:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:54:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T12:54:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T12:54:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T12:54:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 4734d245-46e2-4286-95a3-3cc562d928fa 2024-09-16T12:54:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:54:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:48 2024-09-16T12:54:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:54:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T12:54:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T12:54:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T12:54:49.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:54:48' 2024-09-16T12:54:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T12:54:49.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T12:54:49.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:54:49.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:48' 2024-09-16T12:54:49.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T12:54:49.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T12:54:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:48' 2024-09-16T12:54:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:54:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T12:54:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:54:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:49.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:54:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING7342654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary6e1674f59-5785-4885-8027-0915586882c9up+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary94734d245-46e2-4286-95a3-3cc562d928faup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48' 2024-09-16T12:54:49.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:54:49.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:54:49.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:54:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:54:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T12:54:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:49.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:54:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:49.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:54:49.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:54:49.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-16T12:54:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:54:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:54:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-16T12:54:49.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:54:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:48' 2024-09-16T12:54:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-16 12:54:48' 2024-09-16T12:54:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:54:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T12:54:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:54:48' 2024-09-16T12:54:49.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:54:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-16T12:54:49.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:54:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:54:49.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-16T12:54:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:48' 2024-09-16T12:54:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T12:54:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:54:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T12:54:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:54:49.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:54:49.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:49.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-16T12:54:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:54:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:54:49.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-16T12:54:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING7342654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary6e1674f59-5785-4885-8027-0915586882c9up+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+stoppedlocal image is primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}2024-09-16 12:54:48image-primary94734d245-46e2-4286-95a3-3cc562d928faup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 12:54:48cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 12:54:48' 2024-09-16T12:54:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:54:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T12:54:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:54:48' 2024-09-16T12:54:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T12:54:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T12:54:49.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T12:54:49.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T12:54:49.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T12:54:49.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:49.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:54:49.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-16T12:54:49.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:54:49.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:54:49.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:54:49.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T12:54:49.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-16T12:54:50.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:54:50.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:54:50.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:54:50.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:54:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:54:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:54:50.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary3 2024-09-16T12:54:50.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:54:50.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T12:54:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:54:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:54:50.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:54:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary7 2024-09-16T12:54:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:54:50.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:54:50.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary1 2024-09-16T12:54:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary9 2024-09-16T12:54:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary4 2024-09-16T12:54:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary5 2024-09-16T12:54:50.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:54:50.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary2 2024-09-16T12:54:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd3 2024-09-16T12:54:50.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:54:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd3 2024-09-16T12:54:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd4 2024-09-16T12:54:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:54:50.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd4 2024-09-16T12:54:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd5 2024-09-16T12:54:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:54:50.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd5 2024-09-16T12:54:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd6 2024-09-16T12:54:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:54:51.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd6 2024-09-16T12:54:51.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd7 2024-09-16T12:54:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:54:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd7 2024-09-16T12:54:51.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd8 2024-09-16T12:54:51.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:54:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd8 2024-09-16T12:54:51.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd9 2024-09-16T12:54:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T12:54:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd9 2024-09-16T12:55:22.113 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:55:22.098+0000 7f74def64700 -1 reset not still connected to 0x5594556f0270 2024-09-16T12:55:31.620 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:55:31.614+0000 7f315ffe6700 -1 reset not still connected to 0x55c204a4de10 2024-09-16T12:56:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=b92db8d54caf32ca485b93013713f6c0 2024-09-16T12:56:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd2 2024-09-16T12:56:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=0d82bcf3841c016be2df5a6e95c968f7 2024-09-16T12:56:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-09-16T12:56:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=8d5b742d7a269c511a8ae602d7950dec 2024-09-16T12:56:20.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd1 2024-09-16T12:56:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ b92db8d54caf32ca485b93013713f6c0 != \b\9\2\d\b\8\d\5\4\c\a\f\3\2\c\a\4\8\5\b\9\3\0\1\3\7\1\3\f\6\c\0 ]] 2024-09-16T12:56:20.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 0d82bcf3841c016be2df5a6e95c968f7 != \0\d\8\2\b\c\f\3\8\4\1\c\0\1\6\b\e\2\d\f\5\a\6\e\9\5\c\9\6\8\f\7 ]] 2024-09-16T12:56:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 8d5b742d7a269c511a8ae602d7950dec != \8\d\5\b\7\4\2\d\7\a\2\6\9\c\5\1\1\a\8\a\e\6\0\2\d\7\9\5\0\d\e\c ]] 2024-09-16T12:56:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=5d935b7e472014244095439c5d2e79b6 2024-09-16T12:56:23.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd3 2024-09-16T12:56:23.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 5d935b7e472014244095439c5d2e79b6 != \5\d\9\3\5\b\7\e\4\7\2\0\1\4\2\4\4\0\9\5\4\3\9\c\5\d\2\e\7\9\b\6 ]] 2024-09-16T12:56:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=a8a29ac1763dc1bf0d7e751cac12b6c2 2024-09-16T12:56:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd8 2024-09-16T12:56:24.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=c0e960550a19042eb11a659270978bde 2024-09-16T12:56:24.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd4 2024-09-16T12:56:24.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=8d8245fea0e9ec7342f3b3695d600ad9 2024-09-16T12:56:24.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd9 2024-09-16T12:56:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=f7c5d2d5fc7a952da5632bdef4b4a1dd 2024-09-16T12:56:25.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd5 2024-09-16T12:56:25.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ a8a29ac1763dc1bf0d7e751cac12b6c2 != \a\8\a\2\9\a\c\1\7\6\3\d\c\1\b\f\0\d\7\e\7\5\1\c\a\c\1\2\b\6\c\2 ]] 2024-09-16T12:56:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ c0e960550a19042eb11a659270978bde != \c\0\e\9\6\0\5\5\0\a\1\9\0\4\2\e\b\1\1\a\6\5\9\2\7\0\9\7\8\b\d\e ]] 2024-09-16T12:56:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=e90baeb1de83703427b62ae5fd2ddab8 2024-09-16T12:56:25.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd7 2024-09-16T12:56:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=8ccd2b1d27f7c9d89bf84d7adb22c6af 2024-09-16T12:56:25.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd6 2024-09-16T12:56:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 8d8245fea0e9ec7342f3b3695d600ad9 != \8\d\8\2\4\5\f\e\a\0\e\9\e\c\7\3\4\2\f\3\b\3\6\9\5\d\6\0\0\a\d\9 ]] 2024-09-16T12:56:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:56:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 70217 2024-09-16T12:56:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:56:25.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 70219 2024-09-16T12:56:25.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ f7c5d2d5fc7a952da5632bdef4b4a1dd != \f\7\c\5\d\2\d\5\f\c\7\a\9\5\2\d\a\5\6\3\2\b\d\e\f\4\b\4\a\1\d\d ]] 2024-09-16T12:56:26.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ e90baeb1de83703427b62ae5fd2ddab8 != \e\9\0\b\a\e\b\1\d\e\8\3\7\0\3\4\2\7\b\6\2\a\e\5\f\d\2\d\d\a\b\8 ]] 2024-09-16T12:56:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 8ccd2b1d27f7c9d89bf84d7adb22c6af != \8\c\c\d\2\b\1\d\2\7\f\7\c\9\d\8\9\b\f\8\4\d\7\a\d\b\2\2\c\6\a\f ]] 2024-09-16T12:56:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:56:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 70221 2024-09-16T12:56:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:56:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 70223 2024-09-16T12:56:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:56:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 70224 2024-09-16T12:56:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:56:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 70226 2024-09-16T12:56:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:56:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 70227 2024-09-16T12:56:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:56:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 70230 2024-09-16T12:56:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T12:56:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 70231 2024-09-16T12:56:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T12:56:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:56:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary1 2024-09-16T12:56:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-16T12:56:26.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:56:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:56:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:56:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:56:26.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:56:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-16T12:56:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:56:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:56:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:56:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:56:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:56:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T12:56:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T12:56:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: c1088e42-e596-4d28-9923-42d6045d5571 2024-09-16T12:56:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:56:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72364032.0,"last_snapshot_bytes":72364032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"} 2024-09-16T12:56:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:56:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:56:23 2024-09-16T12:56:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:56:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:56:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:56:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:56:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:56:23' 2024-09-16T12:56:27.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:56:27.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:56:27.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:56:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72364032.0,"last_snapshot_bytes":72364032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"} 2024-09-16T12:56:27.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:56:27.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:56:27.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:56:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:56:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:56:27.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:56:27.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:56:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:56:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:56:23' 2024-09-16T12:56:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-16 12:56:23' 2024-09-16T12:56:27.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:56:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:56:23' 2024-09-16T12:56:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:56:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:56:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:56:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1c1088e42-e596-4d28-9923-42d6045d5571up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72364032.0,"last_snapshot_bytes":72364032,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:23image-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:23image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70275072.0,"last_snapshot_bytes":70275072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:23image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72396800.0,"last_snapshot_bytes":72396800,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:23image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":104235008.0,"last_snapshot_bytes":104235008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:23image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70057984.0,"last_snapshot_bytes":70057984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:23image-primary6e1674f59-5785-4885-8027-0915586882c9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70381568.0,"last_snapshot_bytes":70381568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:23image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63877120.0,"last_snapshot_bytes":63877120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:23image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72368128.0,"last_snapshot_bytes":72368128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:23image-primary94734d245-46e2-4286-95a3-3cc562d928faup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:23' 2024-09-16T12:56:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:56:27.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T12:56:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:56:23' 2024-09-16T12:56:28.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T12:56:28.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:56:28.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:56:28.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:56:28.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:56:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-16T12:56:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:56:28.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:56:28.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-16T12:56:28.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-16T12:56:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:56:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-16T12:56:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:56:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:56:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:56:29.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary1 2024-09-16T12:56:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary1 2024-09-16T12:56:30.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:56:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary2 2024-09-16T12:56:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-16T12:56:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:56:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:56:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:56:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:56:30.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:56:31.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-16T12:56:31.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:31.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:31.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:56:31.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:31.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:56:31.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:56:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:56:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:56:31.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T12:56:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T12:56:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 8d0e5cc3-e3f8-4977-9e72-a3c1492e4e85 2024-09-16T12:56:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:56:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70275072.0,"last_snapshot_bytes":70275072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"} 2024-09-16T12:56:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:56:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:56:23 2024-09-16T12:56:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:56:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:56:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:56:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:56:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:56:28' 2024-09-16T12:56:31.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:56:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:56:31.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:56:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70275072.0,"last_snapshot_bytes":70275072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"} 2024-09-16T12:56:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:56:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:56:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:56:31.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:56:31.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:56:31.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:56:31.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:56:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:56:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:56:23' 2024-09-16T12:56:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-16 12:56:23' 2024-09-16T12:56:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:56:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:56:23' 2024-09-16T12:56:31.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:56:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:56:31.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:56:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK942544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:28image-primary28d0e5cc3-e3f8-4977-9e72-a3c1492e4e85up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70275072.0,"last_snapshot_bytes":70275072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:28image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72396800.0,"last_snapshot_bytes":72396800,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:28image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":104235008.0,"last_snapshot_bytes":104235008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:28image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70057984.0,"last_snapshot_bytes":70057984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:28image-primary6e1674f59-5785-4885-8027-0915586882c9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70381568.0,"last_snapshot_bytes":70381568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:28image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63877120.0,"last_snapshot_bytes":63877120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:28image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72368128.0,"last_snapshot_bytes":72368128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:28image-primary94734d245-46e2-4286-95a3-3cc562d928faup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:28' 2024-09-16T12:56:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:56:31.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T12:56:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:56:23' 2024-09-16T12:56:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T12:56:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:56:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:56:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:56:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:56:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-16T12:56:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:56:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:56:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-16T12:56:31.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-16T12:56:32.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:56:32.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-16T12:56:32.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:32.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:32.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:56:32.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:56:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:56:32.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary2 2024-09-16T12:56:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:32.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary2 2024-09-16T12:56:32.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:56:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary3 2024-09-16T12:56:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-16T12:56:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:56:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:56:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:56:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:56:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:56:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-16T12:56:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:56:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:56:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:56:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:56:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:56:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T12:56:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T12:56:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: fec07e41-ee3f-4c53-baf9-b248297f6077 2024-09-16T12:56:33.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:56:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72396800.0,"last_snapshot_bytes":72396800,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"} 2024-09-16T12:56:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:56:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:56:23 2024-09-16T12:56:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:56:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:56:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:56:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:56:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:56:33' 2024-09-16T12:56:33.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:56:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:56:33.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:56:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72396800.0,"last_snapshot_bytes":72396800,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"} 2024-09-16T12:56:33.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:56:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:56:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:56:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:56:33.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:56:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:56:33.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:56:33.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:56:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:56:23' 2024-09-16T12:56:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-16 12:56:23' 2024-09-16T12:56:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:56:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:56:23' 2024-09-16T12:56:33.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:56:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:56:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:56:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK842544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:33image-primary3fec07e41-ee3f-4c53-baf9-b248297f6077up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72396800.0,"last_snapshot_bytes":72396800,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:33image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":104235008.0,"last_snapshot_bytes":104235008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:33image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70057984.0,"last_snapshot_bytes":70057984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:33image-primary6e1674f59-5785-4885-8027-0915586882c9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70381568.0,"last_snapshot_bytes":70381568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:33image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63877120.0,"last_snapshot_bytes":63877120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:33image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72368128.0,"last_snapshot_bytes":72368128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:33image-primary94734d245-46e2-4286-95a3-3cc562d928faup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:33' 2024-09-16T12:56:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:56:34.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T12:56:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:56:23' 2024-09-16T12:56:34.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T12:56:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:56:34.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:56:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:56:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:56:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-16T12:56:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:56:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:56:34.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-16T12:56:34.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-16T12:56:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:56:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-16T12:56:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:56:35.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:56:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:56:35.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary3 2024-09-16T12:56:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:35.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary3 2024-09-16T12:56:35.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:56:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary4 2024-09-16T12:56:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-16T12:56:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:56:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:56:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:56:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:56:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:56:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-16T12:56:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:56:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:36.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:56:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:56:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:56:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:56:36.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T12:56:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T12:56:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: dd6269fd-f848-44f5-b3e7-af6943c54ccb 2024-09-16T12:56:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:56:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":104235008.0,"last_snapshot_bytes":104235008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"} 2024-09-16T12:56:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:56:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:56:23 2024-09-16T12:56:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:56:36.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:56:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:56:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:56:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:56:33' 2024-09-16T12:56:36.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:56:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:56:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:56:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":104235008.0,"last_snapshot_bytes":104235008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"} 2024-09-16T12:56:36.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:56:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:56:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:56:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:56:36.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:56:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:56:36.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:56:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:56:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:56:23' 2024-09-16T12:56:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-16 12:56:23' 2024-09-16T12:56:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:36.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:56:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:56:23' 2024-09-16T12:56:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:56:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:56:36.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:56:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK742544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:33image-primary4dd6269fd-f848-44f5-b3e7-af6943c54ccbup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":104235008.0,"last_snapshot_bytes":104235008,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:33image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70057984.0,"last_snapshot_bytes":70057984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:33image-primary6e1674f59-5785-4885-8027-0915586882c9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70381568.0,"last_snapshot_bytes":70381568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:33image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63877120.0,"last_snapshot_bytes":63877120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:33image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72368128.0,"last_snapshot_bytes":72368128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:33image-primary94734d245-46e2-4286-95a3-3cc562d928faup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:33' 2024-09-16T12:56:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:56:37.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T12:56:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:56:23' 2024-09-16T12:56:37.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T12:56:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:56:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:56:37.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:56:37.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:56:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-16T12:56:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:56:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:56:37.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-16T12:56:37.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-16T12:56:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:56:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-16T12:56:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:56:38.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:56:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:56:38.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary4 2024-09-16T12:56:38.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:38.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary4 2024-09-16T12:56:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:56:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary5 2024-09-16T12:56:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-16T12:56:38.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:56:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:56:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:56:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:56:38.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:56:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-16T12:56:39.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:56:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:56:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:56:39.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:56:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:56:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T12:56:39.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T12:56:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: aedce2c3-b936-41c9-95f9-3b2be5e43d04 2024-09-16T12:56:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:56:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70057984.0,"last_snapshot_bytes":70057984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"} 2024-09-16T12:56:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:56:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:56:23 2024-09-16T12:56:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:56:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:56:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:56:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:56:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:56:38' 2024-09-16T12:56:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:56:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:56:39.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:56:39.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70057984.0,"last_snapshot_bytes":70057984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"} 2024-09-16T12:56:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:56:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:56:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:56:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:56:39.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:56:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:56:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:56:39.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:56:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:56:23' 2024-09-16T12:56:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-16 12:56:23' 2024-09-16T12:56:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:56:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:56:23' 2024-09-16T12:56:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:56:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:56:39.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:56:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK642544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:38image-primary5aedce2c3-b936-41c9-95f9-3b2be5e43d04up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70057984.0,"last_snapshot_bytes":70057984,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:38image-primary6e1674f59-5785-4885-8027-0915586882c9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70381568.0,"last_snapshot_bytes":70381568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:38image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63877120.0,"last_snapshot_bytes":63877120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:38image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72368128.0,"last_snapshot_bytes":72368128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:38image-primary94734d245-46e2-4286-95a3-3cc562d928faup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:38' 2024-09-16T12:56:40.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:56:40.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T12:56:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:56:23' 2024-09-16T12:56:40.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T12:56:40.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:56:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:56:40.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:56:40.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:56:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-16T12:56:40.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:56:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:56:40.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-16T12:56:40.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-16T12:56:41.537 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:56:41.527+0000 7f4071c2f700 -1 reset not still connected to 0x563591d7d040 2024-09-16T12:56:41.538 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:56:41.527+0000 7f4071c2f700 -1 reset not still connected to 0x5635a3d41040 2024-09-16T12:56:41.542 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:56:41.531+0000 7f4071c2f700 -1 reset not still connected to 0x5635ad3ed380 2024-09-16T12:56:41.543 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T12:56:41.531+0000 7f4071c2f700 -1 reset not still connected to 0x5635adb39110 2024-09-16T12:56:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:56:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-16T12:56:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:56:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:56:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:56:41.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary5 2024-09-16T12:56:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary5 2024-09-16T12:56:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:56:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary6 2024-09-16T12:56:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-16T12:56:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:56:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:56:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:56:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:56:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:56:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-16T12:56:42.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:56:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:56:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:56:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:56:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:56:42.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T12:56:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T12:56:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: e1674f59-5785-4885-8027-0915586882c9 2024-09-16T12:56:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:56:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70381568.0,"last_snapshot_bytes":70381568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"} 2024-09-16T12:56:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:56:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:56:23 2024-09-16T12:56:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:56:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:56:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:56:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:56:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:56:38' 2024-09-16T12:56:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:56:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:56:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:56:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70381568.0,"last_snapshot_bytes":70381568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"} 2024-09-16T12:56:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:56:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:56:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:56:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:56:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:56:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:56:42.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:56:42.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:56:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:56:23' 2024-09-16T12:56:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-16 12:56:23' 2024-09-16T12:56:42.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:56:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:56:23' 2024-09-16T12:56:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:56:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:56:42.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:56:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK542544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:38image-primary6e1674f59-5785-4885-8027-0915586882c9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70381568.0,"last_snapshot_bytes":70381568,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:38image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63877120.0,"last_snapshot_bytes":63877120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:38image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72368128.0,"last_snapshot_bytes":72368128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:38image-primary94734d245-46e2-4286-95a3-3cc562d928faup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:38' 2024-09-16T12:56:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:56:43.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T12:56:43.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:56:23' 2024-09-16T12:56:43.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T12:56:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:56:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:56:43.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:56:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:56:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-16T12:56:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:56:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:56:43.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-16T12:56:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-16T12:56:44.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:56:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-16T12:56:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:56:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:56:44.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary6 2024-09-16T12:56:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:56:44.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary6 2024-09-16T12:56:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:56:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary7 2024-09-16T12:56:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-16T12:56:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:44.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:56:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:56:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:56:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:56:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:56:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-16T12:56:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:56:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:56:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:56:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:56:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:56:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T12:56:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T12:56:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: afa1056f-626c-48ac-b10e-3d158ee2f57a 2024-09-16T12:56:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:56:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63877120.0,"last_snapshot_bytes":63877120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"} 2024-09-16T12:56:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:56:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:56:23 2024-09-16T12:56:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:56:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:56:45.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:56:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:56:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:56:43' 2024-09-16T12:56:45.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:56:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:56:45.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:56:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63877120.0,"last_snapshot_bytes":63877120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"} 2024-09-16T12:56:45.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:56:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:56:45.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:56:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:56:45.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:56:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:56:45.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:56:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:56:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:56:23' 2024-09-16T12:56:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-16 12:56:23' 2024-09-16T12:56:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:56:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:56:23' 2024-09-16T12:56:45.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:56:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:56:45.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:56:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK442544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:43image-primary7afa1056f-626c-48ac-b10e-3d158ee2f57aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63877120.0,"last_snapshot_bytes":63877120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:43image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72368128.0,"last_snapshot_bytes":72368128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:43image-primary94734d245-46e2-4286-95a3-3cc562d928faup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:43' 2024-09-16T12:56:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:56:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T12:56:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:56:23' 2024-09-16T12:56:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T12:56:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:56:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:56:46.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:56:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:56:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-16T12:56:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:56:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:56:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-16T12:56:46.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-16T12:56:47.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:56:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-16T12:56:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:56:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:56:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:56:47.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary7 2024-09-16T12:56:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:47.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary7 2024-09-16T12:56:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:56:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary8 2024-09-16T12:56:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-16T12:56:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:56:47.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:56:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:56:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:56:47.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:56:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-16T12:56:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:56:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:56:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:56:48.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:56:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:56:48.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T12:56:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T12:56:48.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 729e14b6-bf4a-4677-9904-ec7f7484589d 2024-09-16T12:56:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:56:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72368128.0,"last_snapshot_bytes":72368128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"} 2024-09-16T12:56:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:56:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:56:23 2024-09-16T12:56:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:56:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:56:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:56:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:56:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:56:43' 2024-09-16T12:56:48.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:56:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:56:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:56:48.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72368128.0,"last_snapshot_bytes":72368128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"} 2024-09-16T12:56:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:56:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:56:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:56:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:56:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:56:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:56:48.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:56:48.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:56:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:56:23' 2024-09-16T12:56:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-16 12:56:23' 2024-09-16T12:56:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:56:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:56:23' 2024-09-16T12:56:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:56:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:56:48.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:56:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK342544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:48image-primary8729e14b6-bf4a-4677-9904-ec7f7484589dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72368128.0,"last_snapshot_bytes":72368128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:48image-primary94734d245-46e2-4286-95a3-3cc562d928faup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:48' 2024-09-16T12:56:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:56:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T12:56:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:56:23' 2024-09-16T12:56:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T12:56:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:56:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:56:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:56:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:56:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-16T12:56:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:56:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:56:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-16T12:56:48.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-16T12:56:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:56:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-16T12:56:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:56:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:56:49.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:56:49.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary8 2024-09-16T12:56:49.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:49.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary8 2024-09-16T12:56:49.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:56:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary9 2024-09-16T12:56:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-16T12:56:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:56:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:56:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:56:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:56:49.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:56:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-16T12:56:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:56:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:56:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:56:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:56:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:56:50.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T12:56:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T12:56:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 4734d245-46e2-4286-95a3-3cc562d928fa 2024-09-16T12:56:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:56:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"} 2024-09-16T12:56:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:56:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:56:23 2024-09-16T12:56:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:56:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:56:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:56:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:56:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:56:48' 2024-09-16T12:56:50.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:56:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:56:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:56:50.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"} 2024-09-16T12:56:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:56:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:56:50.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:56:50.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:56:50.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:56:50.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:56:50.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:56:50.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:56:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:56:23' 2024-09-16T12:56:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-16 12:56:23' 2024-09-16T12:56:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:56:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:56:23' 2024-09-16T12:56:50.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:56:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:56:50.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:56:51.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK242544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:48image-primary94734d245-46e2-4286-95a3-3cc562d928faup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491290,"remote_snapshot_timestamp":1726491290,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:48' 2024-09-16T12:56:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:56:51.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T12:56:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:56:23' 2024-09-16T12:56:51.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T12:56:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:56:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:56:51.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:56:51.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:56:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-16T12:56:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:56:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:51.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:56:51.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-16T12:56:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-16T12:56:52.314 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:56:52.303+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1748f0 2024-09-16T12:56:52.315 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:56:52.303+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174d00 2024-09-16T12:56:52.315 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:56:52.303+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T12:56:52.315 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:56:52.303+0000 7ff4f6946700 -1 reset not still connected to 0x55756ada0a90 2024-09-16T12:56:52.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:56:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-16T12:56:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:56:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:56:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:56:52.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary9 2024-09-16T12:56:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary9 2024-09-16T12:56:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:56:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary10 2024-09-16T12:56:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-16T12:56:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:56:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:56:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:56:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:56:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T12:56:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-16T12:56:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:56:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T12:56:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T12:56:53.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:56:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:56:53.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T12:56:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T12:56:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: c82934ef-5cb6-459e-a80f-ec8e46fbc42a 2024-09-16T12:56:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T12:56:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"} 2024-09-16T12:56:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T12:56:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:56:53 2024-09-16T12:56:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T12:56:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T12:56:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T12:56:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T12:56:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 12:56:53' 2024-09-16T12:56:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T12:56:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T12:56:53.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T12:56:53.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"} 2024-09-16T12:56:53.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T12:56:53.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:56:53.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T12:56:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:56:53.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T12:56:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T12:56:53.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T12:56:53.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T12:56:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:56:53' 2024-09-16T12:56:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-16 12:56:53' 2024-09-16T12:56:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:56:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T12:56:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 12:56:53' 2024-09-16T12:56:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T12:56:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T12:56:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T12:56:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK142544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10c82934ef-5cb6-459e-a80f-ec8e46fbc42aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70074368.0,"last_snapshot_bytes":70074368,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491286,"remote_snapshot_timestamp":1726491286,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 12:56:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 12:56:53' 2024-09-16T12:56:54.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T12:56:54.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T12:56:54.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 12:56:53' 2024-09-16T12:56:54.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T12:56:54.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T12:56:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T12:56:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T12:56:54.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T12:56:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-16T12:56:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T12:56:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:56:54.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-16T12:56:54.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-16T12:56:55.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T12:56:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-16T12:56:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:56:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T12:56:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T12:56:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary10 2024-09-16T12:56:55.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T12:56:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary10 2024-09-16T12:56:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..10} 2024-09-16T12:56:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS=() 2024-09-16T12:56:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS=() 2024-09-16T12:56:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS=() 2024-09-16T12:56:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS=() 2024-09-16T12:56:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ RET=0 2024-09-16T12:56:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:56:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary1 2024-09-16T12:56:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary1 2024-09-16T12:56:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-16T12:56:55.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:56:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:56:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:56:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-16T12:56:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:56:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:56:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:56:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:56:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-16T12:56:55.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:56:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-16T12:56:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:56:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:56:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-16T12:56:56.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:56:56.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-16T12:56:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary1': 2024-09-16T12:56:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:56:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:56:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:56:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 235b2f1cd2b0 2024-09-16T12:56:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.235b2f1cd2b0 2024-09-16T12:56:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:56:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:56:56.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:56:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:56:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:56:55 2024 2024-09-16T12:56:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:56:55 2024 2024-09-16T12:56:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:56:55 2024 2024-09-16T12:56:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:56:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:56:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: a9797a82-534a-4a13-8444-a14e2d2586a3 2024-09-16T12:56:56.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:56:56.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:56:56.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary1 2024-09-16T12:56:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd0 2024-09-16T12:56:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:56:56.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd0 2024-09-16T12:56:56.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:56:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:56:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:56:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 1494aca5-7d12-4d59-9b2f-50d4a53792db 2024-09-16T12:56:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:56:57.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:56:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:56:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:56:57.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:56:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:56:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:56:57.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:56:57.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary1 2024-09-16T12:56:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd0 test-primary1 2024-09-16T12:56:57.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:56:57.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary1 2024-09-16T12:56:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:56:57.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-16T12:56:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:56:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:56:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary1 2024-09-16T12:56:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:56:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary2 2024-09-16T12:56:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:56:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary2 2024-09-16T12:56:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:56:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-16T12:56:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:57.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary1 2024-09-16T12:56:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-16T12:56:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:56:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:56:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:56:57.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:56:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:56:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-16T12:56:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:57.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:56:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:56:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:56:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:56:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-16T12:56:57.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:56:57.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-16T12:56:57.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:57.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:56:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:56:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-16T12:56:57.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:56:57.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-16T12:56:58.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2634 2024-09-16T12:56:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:56:58.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:56:58.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-16T12:56:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary2': 2024-09-16T12:56:58.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:56:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:56:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:56:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 236b19abdd29 2024-09-16T12:56:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.236b19abdd29 2024-09-16T12:56:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:56:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:56:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:56:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:56:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:56:57 2024 2024-09-16T12:56:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:56:57 2024 2024-09-16T12:56:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:56:57 2024 2024-09-16T12:56:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:56:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:56:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 196c49a1-a2e7-49aa-a4ea-cf8227ec3d71 2024-09-16T12:56:58.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:56:58.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:56:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary2 2024-09-16T12:56:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd1 2024-09-16T12:56:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:56:58.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd1 2024-09-16T12:56:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:56:59.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/26214402101248/2621440 done 2024-09-16T12:56:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:56:59.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 9833a1fe-b64b-4b39-955f-cd77661a779b 2024-09-16T12:56:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:56:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:56:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:56:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:56:59.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:56:59.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:56:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:56:59.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:56:59.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary2 2024-09-16T12:56:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd1 test-primary2 2024-09-16T12:56:59.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:56:59.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary2 2024-09-16T12:56:59.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:56:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-16T12:56:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:56:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary2 2024-09-16T12:56:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:56:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary3 2024-09-16T12:56:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary3 2024-09-16T12:56:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary2 2024-09-16T12:56:59.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-16T12:56:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-16T12:56:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:56:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:56:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:56:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-16T12:56:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:56:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:56:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:56:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:56:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:56:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:56:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-16T12:56:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:56:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:56:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:56:59.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:56:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:56:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-16T12:56:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:56:59.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:56:59.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:56:59.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:56:59.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-16T12:56:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:56:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-16T12:57:00.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2638 2024-09-16T12:57:00.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:57:00.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:00.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-16T12:57:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary3': 2024-09-16T12:57:00.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:57:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:57:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:57:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 237be9cc30a8 2024-09-16T12:57:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.237be9cc30a8 2024-09-16T12:57:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:57:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:57:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:57:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:57:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:56:59 2024 2024-09-16T12:57:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:56:59 2024 2024-09-16T12:57:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:56:59 2024 2024-09-16T12:57:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:57:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:57:00.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ff 2024-09-16T12:57:00.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:57:00.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:57:00.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary3 2024-09-16T12:57:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd2 2024-09-16T12:57:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:57:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd2 2024-09-16T12:57:00.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:57:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:57:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:57:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: d8aec794-a67e-45ad-a9b5-778af6b1fb12 2024-09-16T12:57:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:57:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:57:00.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:57:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:57:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:57:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:57:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:57:01.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:57:01.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary3 2024-09-16T12:57:01.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd2 test-primary3 2024-09-16T12:57:01.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:57:01.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary3 2024-09-16T12:57:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:57:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-16T12:57:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:57:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:57:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:57:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:57:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:57:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary3 2024-09-16T12:57:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:57:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary4 2024-09-16T12:57:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary4 2024-09-16T12:57:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-16T12:57:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary3 2024-09-16T12:57:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-16T12:57:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:01.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:57:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:01.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:57:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:57:01.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-16T12:57:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:57:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:57:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:57:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:57:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:01.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-16T12:57:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:57:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-16T12:57:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:01.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:57:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:57:01.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-16T12:57:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:57:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-16T12:57:01.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:01.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:57:01.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:01.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:01.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:57:01.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:57:01.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2641 2024-09-16T12:57:01.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:02.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:57:02.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-16T12:57:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary4': 2024-09-16T12:57:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:57:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:57:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:57:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 238aa3d4329 2024-09-16T12:57:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.238aa3d4329 2024-09-16T12:57:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:57:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:57:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:57:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:57:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:57:01 2024 2024-09-16T12:57:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:57:01 2024 2024-09-16T12:57:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:57:01 2024 2024-09-16T12:57:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:57:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:57:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 90dd28e8-9a26-4b71-8b37-c0cde9f6ea0e 2024-09-16T12:57:02.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:57:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:57:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary4 2024-09-16T12:57:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd3 2024-09-16T12:57:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:57:02.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd3 2024-09-16T12:57:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:57:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:57:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:57:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 48df4e2a-913d-4a92-ac74-2f8a048ebea6 2024-09-16T12:57:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:57:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:57:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:57:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:57:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:57:03.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): Snapshot ID: 2643 2024-09-16T12:57:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T12:57:03.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:03.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:57:03.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:57:03.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary4 2024-09-16T12:57:03.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd3 test-primary4 2024-09-16T12:57:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:57:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:57:03.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:57:03.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:57:03.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary4 2024-09-16T12:57:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:57:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-16T12:57:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:57:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:57:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:57:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:57:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:57:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:57:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary5 2024-09-16T12:57:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary5 2024-09-16T12:57:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-16T12:57:03.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:57:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:57:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:57:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-16T12:57:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:57:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:57:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:57:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:57:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-16T12:57:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary4 2024-09-16T12:57:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary4 2024-09-16T12:57:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-16T12:57:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:57:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-16T12:57:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:57:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:57:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-16T12:57:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:57:03.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-16T12:57:04.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2646 2024-09-16T12:57:04.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:57:04.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-16T12:57:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:57:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:57:04.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:57:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary5': 2024-09-16T12:57:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:57:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:57:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:57:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 239f7eda0fe7 2024-09-16T12:57:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.239f7eda0fe7 2024-09-16T12:57:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:57:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:57:04.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:57:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:57:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:57:03 2024 2024-09-16T12:57:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:57:03 2024 2024-09-16T12:57:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:57:03 2024 2024-09-16T12:57:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:57:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:57:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50 2024-09-16T12:57:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:57:04.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:57:04.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary5 2024-09-16T12:57:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd4 2024-09-16T12:57:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:57:04.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd4 2024-09-16T12:57:04.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:57:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/26214401576960/2621440 done 2024-09-16T12:57:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:57:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 65a7c635-4ae3-47a5-be2a-de442d6e4d8c 2024-09-16T12:57:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:57:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:57:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:57:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:57:05.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:57:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:57:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:57:05.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:57:05.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary5 2024-09-16T12:57:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd4 test-primary5 2024-09-16T12:57:05.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2645 2024-09-16T12:57:05.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:05.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:57:05.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary5 2024-09-16T12:57:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:57:05.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-16T12:57:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:57:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:57:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:57:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:57:05.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:57:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:57:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary5 2024-09-16T12:57:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary6 2024-09-16T12:57:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary6 2024-09-16T12:57:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-16T12:57:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary5 2024-09-16T12:57:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-16T12:57:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:05.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:57:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:57:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:57:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-16T12:57:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:57:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:57:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:57:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:57:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:05.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-16T12:57:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:57:05.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-16T12:57:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:57:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:57:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-16T12:57:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:57:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-16T12:57:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:57:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:06.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:57:06.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:57:06.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2651 2024-09-16T12:57:06.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2654 2024-09-16T12:57:06.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:06.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:57:06.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-16T12:57:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary6': 2024-09-16T12:57:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:57:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:57:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:57:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 23b287e34d8b 2024-09-16T12:57:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.23b287e34d8b 2024-09-16T12:57:06.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:57:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:57:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:57:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:57:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:57:05 2024 2024-09-16T12:57:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:57:05 2024 2024-09-16T12:57:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:57:05 2024 2024-09-16T12:57:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:57:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:57:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: b18ebe6c-4efe-4743-8aee-94780a9c5a06 2024-09-16T12:57:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:57:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:57:06.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary6 2024-09-16T12:57:06.912 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:57:06.902+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T12:57:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd5 2024-09-16T12:57:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:57:06.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd5 2024-09-16T12:57:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:57:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/26214401052672/2621440 done 2024-09-16T12:57:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:57:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 3dcaa05d-0bee-452f-b923-43fa5b4e0e45 2024-09-16T12:57:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:57:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:57:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:57:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:57:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:57:07.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:57:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 Snapshot ID: 2656 2024-09-16T12:57:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:57:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:57:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:57:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T12:57:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:57:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary6 2024-09-16T12:57:07.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd5 test-primary6 2024-09-16T12:57:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:57:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary6 2024-09-16T12:57:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:57:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-16T12:57:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:57:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:57:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:57:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:57:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:57:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:57:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary7 2024-09-16T12:57:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary7 2024-09-16T12:57:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-16T12:57:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:57:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:57:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:57:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-16T12:57:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:57:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:57:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:57:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:57:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-16T12:57:07.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary6 2024-09-16T12:57:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary6 2024-09-16T12:57:07.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-16T12:57:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:57:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-16T12:57:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:57:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:57:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-16T12:57:07.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:57:07.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-16T12:57:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:08.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:57:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:57:08.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:57:08.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2659 2024-09-16T12:57:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:08.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:57:08.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-16T12:57:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary7': 2024-09-16T12:57:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:57:08.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:57:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:57:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 23c6803ec3d1 2024-09-16T12:57:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.23c6803ec3d1 2024-09-16T12:57:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:57:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:57:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:57:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:57:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:57:07 2024 2024-09-16T12:57:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:57:07 2024 2024-09-16T12:57:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:57:07 2024 2024-09-16T12:57:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:57:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:57:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: f491577d-20f7-4b88-9fb4-e72a02bb468c 2024-09-16T12:57:08.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:57:08.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:57:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary7 2024-09-16T12:57:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd6 2024-09-16T12:57:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:57:08.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd6 2024-09-16T12:57:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:57:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:57:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:57:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 87f53dc6-9c28-4282-98a6-0d8178d55a52 2024-09-16T12:57:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:57:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:57:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:57:08.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:57:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:57:09.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:57:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:57:09.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:57:09.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary7 2024-09-16T12:57:09.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd6 test-primary7 2024-09-16T12:57:09.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2658 2024-09-16T12:57:09.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:57:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:57:09.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:57:09.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:57:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary7 2024-09-16T12:57:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:57:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-16T12:57:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:57:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:57:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:57:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:57:09.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:57:09.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:57:09.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary7 2024-09-16T12:57:09.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary8 2024-09-16T12:57:09.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary8 2024-09-16T12:57:09.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-16T12:57:09.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:09.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary7 2024-09-16T12:57:09.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:09.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:09.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-16T12:57:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:57:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:57:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:57:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-16T12:57:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:57:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:09.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:57:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:57:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:57:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:09.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-16T12:57:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:57:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:57:09.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:57:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:57:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-16T12:57:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:57:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:57:09.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-16T12:57:09.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:57:09.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-16T12:57:10.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2661 2024-09-16T12:57:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:57:10.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:57:10.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:57:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2666 2024-09-16T12:57:10.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:57:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:10.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-16T12:57:10.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary8': 2024-09-16T12:57:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:57:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:57:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:57:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 23d81cea885b 2024-09-16T12:57:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.23d81cea885b 2024-09-16T12:57:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:57:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:57:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:57:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:57:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:57:09 2024 2024-09-16T12:57:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:57:09 2024 2024-09-16T12:57:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:57:09 2024 2024-09-16T12:57:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:57:10.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:57:10.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: c7162764-8ecd-41dd-bf00-31141188bb23 2024-09-16T12:57:10.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:57:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:57:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary8 2024-09-16T12:57:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd7 2024-09-16T12:57:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:57:11.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd7 2024-09-16T12:57:11.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:57:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:57:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:57:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: bc67bd42-315b-4903-b2b3-7fe55d7a380f 2024-09-16T12:57:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:57:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:57:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:57:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:57:11.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:57:11.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:57:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 Snapshot ID: 2667 2024-09-16T12:57:11.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:57:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:57:11.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:57:11.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2663 2024-09-16T12:57:11.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T12:57:11.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:57:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary8 2024-09-16T12:57:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd7 test-primary8 2024-09-16T12:57:11.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:57:11.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary8 2024-09-16T12:57:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:57:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:57:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:57:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary9 2024-09-16T12:57:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary9 2024-09-16T12:57:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-16T12:57:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary8 2024-09-16T12:57:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:11.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-16T12:57:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:57:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:57:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:57:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-16T12:57:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary8 2024-09-16T12:57:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-16T12:57:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:57:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:57:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:57:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:57:11.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-16T12:57:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:57:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:57:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:57:11.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:57:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:57:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-16T12:57:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:11.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:57:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:57:11.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-16T12:57:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:57:11.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-16T12:57:12.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:57:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:57:12.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:57:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2672 2024-09-16T12:57:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:12.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:57:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2676 2024-09-16T12:57:12.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:12.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-16T12:57:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary9': 2024-09-16T12:57:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:57:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:57:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:57:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 23ef99ad17d6 2024-09-16T12:57:12.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.23ef99ad17d6 2024-09-16T12:57:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:57:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:57:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:57:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:57:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:57:11 2024 2024-09-16T12:57:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:57:11 2024 2024-09-16T12:57:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:57:11 2024 2024-09-16T12:57:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:57:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:57:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 18c6fc24-69f8-4190-a25c-6b81712dda2c 2024-09-16T12:57:12.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:57:12.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:57:12.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary9 2024-09-16T12:57:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd8 2024-09-16T12:57:13.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:57:13.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd8 2024-09-16T12:57:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:57:13.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2674 2024-09-16T12:57:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:57:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:57:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 8bf75c3b-3281-414a-8429-cfcab0185fc6 2024-09-16T12:57:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:57:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:57:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:57:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:57:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:57:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:57:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:57:13.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:57:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:57:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:57:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:57:13.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:57:13.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T12:57:13.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:57:13.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary9 2024-09-16T12:57:13.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd8 test-primary9 2024-09-16T12:57:13.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:57:13.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary9 2024-09-16T12:57:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:57:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-16T12:57:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:57:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:57:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:57:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:57:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:57:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T12:57:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary10 2024-09-16T12:57:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary9 2024-09-16T12:57:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary10 2024-09-16T12:57:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-16T12:57:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary9 2024-09-16T12:57:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-16T12:57:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:57:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:57:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T12:57:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-16T12:57:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:13.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:57:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T12:57:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T12:57:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T12:57:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T12:57:13.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-16T12:57:14.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2678 2024-09-16T12:57:14.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:57:14.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-16T12:57:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T12:57:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-16T12:57:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:57:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T12:57:14.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-16T12:57:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:57:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:57:14.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:57:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:57:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:57:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:57:15.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2681 2024-09-16T12:57:15.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2682 2024-09-16T12:57:15.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2683 2024-09-16T12:57:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T12:57:15.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:15.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:15.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-16T12:57:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary10': 2024-09-16T12:57:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T12:57:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T12:57:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T12:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 2408cf8c85c9 2024-09-16T12:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.2408cf8c85c9 2024-09-16T12:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T12:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T12:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T12:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T12:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 12:57:14 2024 2024-09-16T12:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 12:57:14 2024 2024-09-16T12:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 12:57:14 2024 2024-09-16T12:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T12:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T12:57:15.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7 2024-09-16T12:57:15.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T12:57:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T12:57:15.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary10 2024-09-16T12:57:15.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd9 2024-09-16T12:57:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T12:57:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd9 2024-09-16T12:57:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T12:57:15.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T12:57:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T12:57:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: e409fa56-32e6-4821-817f-a1bc245b918c 2024-09-16T12:57:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T12:57:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T12:57:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:57:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T12:57:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T12:57:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:57:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:57:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:57:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:57:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:57:15.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:57:15.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T12:57:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:57:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:16.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:57:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:57:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 Snapshot ID: 2687 2024-09-16T12:57:16.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T12:57:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T12:57:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary10 2024-09-16T12:57:16.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd9 test-primary10 2024-09-16T12:57:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2688 2024-09-16T12:57:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T12:57:16.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary10 2024-09-16T12:57:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T12:57:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-16T12:57:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:57:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T12:57:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:57:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:16.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary10 2024-09-16T12:57:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:57:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:57:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:57:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 74643 2024-09-16T12:57:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary10 2024-09-16T12:57:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-16T12:57:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T12:57:16.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-16T12:57:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2692 2024-09-16T12:57:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2691 2024-09-16T12:57:17.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:17.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:57:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:57:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:57:17.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2696 2024-09-16T12:57:17.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:18.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2693 2024-09-16T12:57:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:57:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:57:18.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:57:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:57:18.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:57:18.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:57:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:18.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:57:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:57:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:57:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2700 2024-09-16T12:57:19.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:57:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:57:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:57:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:57:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:57:19.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:57:20.094 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:57:20.087+0000 7f74def64700 -1 reset not still connected to 0x55946cfe1ad0 2024-09-16T12:57:20.094 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:57:20.087+0000 7f74def64700 -1 reset not still connected to 0x55946cfe1ba0 2024-09-16T12:57:20.095 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:57:20.087+0000 7f74def64700 -1 reset not still connected to 0x55946d9c6680 2024-09-16T12:57:20.095 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:57:20.087+0000 7f74def64700 -1 reset not still connected to 0x55946d9c6750 2024-09-16T12:57:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:57:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:57:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:57:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2703 2024-09-16T12:57:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:57:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:57:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:57:20.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:20.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2704 2024-09-16T12:57:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:57:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:57:20.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:57:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:20.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2705 2024-09-16T12:57:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:57:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:57:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:57:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2710 2024-09-16T12:57:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2706 2024-09-16T12:57:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:22.041 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:57:22.030+0000 7ff4f6946700 -1 reset not still connected to 0x55753c1b1380 2024-09-16T12:57:22.041 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:57:22.034+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174340 2024-09-16T12:57:22.041 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:57:22.034+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1748f0 2024-09-16T12:57:22.042 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:57:22.034+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174d00 2024-09-16T12:57:22.042 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:57:22.034+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T12:57:22.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2713 2024-09-16T12:57:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2712 2024-09-16T12:57:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:57:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:57:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:57:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2715 2024-09-16T12:57:22.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:22.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:23.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2716 2024-09-16T12:57:23.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:57:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:57:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:57:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:57:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:57:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:57:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:57:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:23.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:57:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:57:24.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2720 2024-09-16T12:57:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:57:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:57:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:57:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:57:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:57:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:57:25.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2725 2024-09-16T12:57:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:57:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:57:25.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:57:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2724 2024-09-16T12:57:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:57:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:57:25.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:57:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2723 2024-09-16T12:57:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:57:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:57:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:57:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:25.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:26.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:57:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:57:26.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:57:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2730 2024-09-16T12:57:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2729 2024-09-16T12:57:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:27.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2731 2024-09-16T12:57:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:57:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:57:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:57:27.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:27.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2732 2024-09-16T12:57:27.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2734 2024-09-16T12:57:27.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:27.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2736 2024-09-16T12:57:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:57:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:57:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:57:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:57:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:57:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:57:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:57:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:57:28.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:57:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:57:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:57:29.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:57:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2740 2024-09-16T12:57:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:57:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:57:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:57:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:57:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:57:30.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:57:30.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2744 2024-09-16T12:57:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2743 2024-09-16T12:57:30.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2745 2024-09-16T12:57:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:30.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:57:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:57:30.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:57:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:57:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:57:30.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:57:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:57:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:57:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:57:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2748 2024-09-16T12:57:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2750 2024-09-16T12:57:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:31.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:57:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:57:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:57:32.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2753 2024-09-16T12:57:32.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2755 2024-09-16T12:57:32.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:32.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2754 2024-09-16T12:57:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:33.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2756 2024-09-16T12:57:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:57:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:57:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:57:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:57:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:57:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:57:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:57:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:57:33.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:57:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2761 2024-09-16T12:57:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:57:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:57:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:57:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:57:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:34.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:57:34.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:57:34.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:57:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:57:35.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:57:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2764 2024-09-16T12:57:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:57:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:57:35.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:57:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2763 2024-09-16T12:57:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2765 2024-09-16T12:57:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:57:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:57:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:57:36.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:36.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:57:36.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:36.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:36.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:57:36.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:57:36.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2770 2024-09-16T12:57:36.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2769 2024-09-16T12:57:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:36.247 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:57:36.242+0000 7ff4f6946700 -1 reset not still connected to 0x55755a1355f0 2024-09-16T12:57:36.247 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T12:57:36.242+0000 7ff4f6946700 -1 reset not still connected to 0x5575649b4f70 2024-09-16T12:57:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:57:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:57:37.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:57:37.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2775 2024-09-16T12:57:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2773 2024-09-16T12:57:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:37.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:37.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2774 2024-09-16T12:57:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:38.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2776 2024-09-16T12:57:38.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:57:38.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:57:38.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:57:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:57:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:57:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:57:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:57:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:38.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:57:38.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:57:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2780 2024-09-16T12:57:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:57:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:39.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:57:39.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:57:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:57:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:57:39.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:57:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2785 2024-09-16T12:57:40.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2784 2024-09-16T12:57:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:40.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:57:40.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:40.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:40.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:57:40.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:57:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:57:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:57:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:57:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2783 2024-09-16T12:57:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:57:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:57:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:57:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:41.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:57:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:57:41.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:57:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2788 2024-09-16T12:57:41.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2790 2024-09-16T12:57:41.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:41.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:57:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:57:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:57:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2794 2024-09-16T12:57:42.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2792 2024-09-16T12:57:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2795 2024-09-16T12:57:42.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:42.608 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:57:42.603+0000 7f74def64700 -1 reset not still connected to 0x55946d9c6680 2024-09-16T12:57:42.612 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:57:42.607+0000 7f74def64700 -1 reset not still connected to 0x55946d9c6750 2024-09-16T12:57:43.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2796 2024-09-16T12:57:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:57:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:57:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:57:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:57:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:57:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:57:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:57:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:57:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:57:44.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2800 2024-09-16T12:57:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:57:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:57:44.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:57:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:57:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:57:44.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:57:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2803 2024-09-16T12:57:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2805 2024-09-16T12:57:45.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2804 2024-09-16T12:57:45.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:45.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:45.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:57:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:57:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:57:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:45.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:57:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:57:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:57:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:57:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:57:45.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:57:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:57:46.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2809 2024-09-16T12:57:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:57:46.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:57:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2810 2024-09-16T12:57:46.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:57:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:57:47.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:57:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2815 2024-09-16T12:57:47.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2814 2024-09-16T12:57:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2813 2024-09-16T12:57:47.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:47.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:47.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:47.220 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T12:57:47.214+0000 7f61b1855700 -1 reset not still connected to 0x557b42999ad0 2024-09-16T12:57:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2817 2024-09-16T12:57:48.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:57:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:57:48.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:57:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:57:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:57:48.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:57:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:57:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:48.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:57:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:57:49.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2820 2024-09-16T12:57:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:57:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:49.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:57:49.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:57:49.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:57:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:57:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:57:50.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2823 2024-09-16T12:57:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:57:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:57:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:57:50.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2825 2024-09-16T12:57:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:57:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:57:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:50.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:57:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:57:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:57:50.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:57:50.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2824 2024-09-16T12:57:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:50.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:51.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:57:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:57:51.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:57:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2830 2024-09-16T12:57:51.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2828 2024-09-16T12:57:51.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:51.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:52.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2831 2024-09-16T12:57:52.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2833 2024-09-16T12:57:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:57:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:57:52.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:57:52.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2834 2024-09-16T12:57:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:52.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:52.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:53.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2836 2024-09-16T12:57:53.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:57:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:57:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:57:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:57:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:57:53.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:57:53.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:53.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:57:53.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:57:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:57:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2841 2024-09-16T12:57:54.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:57:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:54.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:57:54.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:57:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:57:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:57:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:57:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2843 2024-09-16T12:57:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:57:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:57:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:57:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2844 2024-09-16T12:57:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:57:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:57:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:57:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:57:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:57:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:57:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2845 2024-09-16T12:57:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:57:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:57:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:57:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2850 2024-09-16T12:57:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2848 2024-09-16T12:57:56.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:56.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:57:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:57:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:57:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2852 2024-09-16T12:57:57.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2851 2024-09-16T12:57:57.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2853 2024-09-16T12:57:57.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:58.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:57:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:57:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:57:58.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2856 2024-09-16T12:57:58.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:58.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:58.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:57:58.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:58.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:58.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:57:58.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:57:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:58.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:57:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:57:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:57:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2860 2024-09-16T12:57:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:57:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:57:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:57:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:57:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:57:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:57:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:57:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:57:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:57:59.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:58:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:58:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:58:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:58:00.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2863 2024-09-16T12:58:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2865 2024-09-16T12:58:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:58:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:58:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:58:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:58:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:00.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:58:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:58:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:00.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2864 2024-09-16T12:58:00.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:01.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:58:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:58:01.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:58:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2868 2024-09-16T12:58:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2869 2024-09-16T12:58:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:01.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:58:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:02.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:58:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:58:02.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2871 2024-09-16T12:58:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2873 2024-09-16T12:58:02.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:02.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:02.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2872 2024-09-16T12:58:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2876 2024-09-16T12:58:03.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:58:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:58:03.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:58:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:58:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:58:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:58:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:58:03.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:58:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:58:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2880 2024-09-16T12:58:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:58:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:58:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:58:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:58:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:58:04.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:58:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:58:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:05.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:58:05.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:58:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:58:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:58:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:58:05.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2883 2024-09-16T12:58:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2885 2024-09-16T12:58:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:58:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:58:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:58:05.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2884 2024-09-16T12:58:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:58:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:06.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:58:06.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:58:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2889 2024-09-16T12:58:06.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2890 2024-09-16T12:58:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:58:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:58:07.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:58:07.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2893 2024-09-16T12:58:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2891 2024-09-16T12:58:07.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2895 2024-09-16T12:58:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:07.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:07.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:08.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2896 2024-09-16T12:58:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:58:08.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:58:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:58:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:58:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:58:08.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:58:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:58:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:58:08.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:58:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2900 2024-09-16T12:58:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:58:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:58:09.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:58:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:58:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:58:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:58:09.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:58:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:58:10.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:58:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2903 2024-09-16T12:58:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:58:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:58:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:58:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:58:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:58:10.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:58:10.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2904 2024-09-16T12:58:10.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2905 2024-09-16T12:58:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:10.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:58:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:58:11.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:58:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2910 2024-09-16T12:58:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:11.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2909 2024-09-16T12:58:11.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:58:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:58:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:58:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2912 2024-09-16T12:58:12.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2915 2024-09-16T12:58:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2914 2024-09-16T12:58:12.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2916 2024-09-16T12:58:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:58:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:58:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:58:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:58:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:58:13.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:58:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:58:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:58:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:58:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2920 2024-09-16T12:58:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:14.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:58:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:58:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:58:14.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:14.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:58:14.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:14.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:14.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:58:14.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:58:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2925 2024-09-16T12:58:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2923 2024-09-16T12:58:15.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:15.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:15.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2924 2024-09-16T12:58:15.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:15.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:58:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:58:15.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:58:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:58:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:58:15.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:58:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:58:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:58:15.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:58:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2929 2024-09-16T12:58:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:58:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:58:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:58:16.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2930 2024-09-16T12:58:16.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:16.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:17.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2935 2024-09-16T12:58:17.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2933 2024-09-16T12:58:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:58:17.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:58:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:58:17.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2934 2024-09-16T12:58:17.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:17.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2939 2024-09-16T12:58:18.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:58:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:58:18.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:58:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:58:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:58:18.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:58:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:58:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:58:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:58:19.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2940 2024-09-16T12:58:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:58:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:58:19.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:58:19.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:58:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:58:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:58:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2944 2024-09-16T12:58:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2943 2024-09-16T12:58:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:58:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:58:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:20.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:58:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:58:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:58:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:58:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:58:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:58:20.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:58:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2945 2024-09-16T12:58:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:58:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:21.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:58:21.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:58:21.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2950 2024-09-16T12:58:21.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:21.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2949 2024-09-16T12:58:21.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:58:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:58:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:58:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2953 2024-09-16T12:58:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2954 2024-09-16T12:58:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2952 2024-09-16T12:58:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:22.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:22.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:23.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2956 2024-09-16T12:58:23.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:23.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:58:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:58:23.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:58:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:58:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:58:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:58:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:58:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:58:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:58:24.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2960 2024-09-16T12:58:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:58:24.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:58:24.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:58:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:58:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:58:24.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:58:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2963 2024-09-16T12:58:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:58:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:58:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:58:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:58:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:58:25.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:58:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:58:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:58:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:58:25.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2964 2024-09-16T12:58:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2965 2024-09-16T12:58:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:58:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:58:26.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:58:26.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2970 2024-09-16T12:58:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2969 2024-09-16T12:58:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:27.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:58:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:58:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:58:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2972 2024-09-16T12:58:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2971 2024-09-16T12:58:27.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2973 2024-09-16T12:58:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:27.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:27.676 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T12:58:27.671+0000 7f74def64700 -1 reset not still connected to 0x55946d9c6750 2024-09-16T12:58:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2976 2024-09-16T12:58:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:58:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:58:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:58:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:58:28.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:58:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:58:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:58:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:58:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:58:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2981 2024-09-16T12:58:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:29.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:58:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:58:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:58:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:58:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:58:29.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:58:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2983 2024-09-16T12:58:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2985 2024-09-16T12:58:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2984 2024-09-16T12:58:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:30.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:58:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:58:30.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:58:30.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:58:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:30.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:58:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:58:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:30.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:58:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:58:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:58:30.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:58:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:58:31.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:58:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2989 2024-09-16T12:58:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2990 2024-09-16T12:58:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2995 2024-09-16T12:58:32.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:32.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:32.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:58:32.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:32.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:32.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:58:32.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:58:32.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2991 2024-09-16T12:58:32.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2992 2024-09-16T12:58:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:33.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 2996 2024-09-16T12:58:33.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:58:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:58:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:58:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:33.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:58:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:58:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:58:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:58:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:58:33.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:58:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3000 2024-09-16T12:58:34.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:58:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:34.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:34.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:58:34.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:58:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:58:34.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:58:34.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:58:35.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3003 2024-09-16T12:58:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3004 2024-09-16T12:58:35.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:35.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3005 2024-09-16T12:58:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:58:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:58:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:58:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:58:35.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:58:35.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:58:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:58:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:58:35.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:58:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:58:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:58:36.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:58:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3009 2024-09-16T12:58:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:36.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3010 2024-09-16T12:58:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:37.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:58:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:58:37.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:58:37.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3014 2024-09-16T12:58:37.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3015 2024-09-16T12:58:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3013 2024-09-16T12:58:37.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:37.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:58:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:58:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:58:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:58:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:58:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:58:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:58:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:58:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:58:38.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3016 2024-09-16T12:58:38.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3020 2024-09-16T12:58:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:58:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:58:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:58:39.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:58:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:58:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:58:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:58:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:58:40.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:58:40.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:58:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:58:40.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:58:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:58:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:58:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:58:40.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3022 2024-09-16T12:58:40.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3024 2024-09-16T12:58:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3023 2024-09-16T12:58:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:40.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:58:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:58:41.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:58:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3029 2024-09-16T12:58:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3030 2024-09-16T12:58:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:58:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:58:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:58:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3032 2024-09-16T12:58:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3034 2024-09-16T12:58:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:42.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3031 2024-09-16T12:58:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3037 2024-09-16T12:58:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:58:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:58:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:58:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:58:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:58:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:58:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:58:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:58:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:58:44.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3040 2024-09-16T12:58:44.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:58:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:58:44.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:58:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:58:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:58:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:58:45.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3045 2024-09-16T12:58:45.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3044 2024-09-16T12:58:45.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:45.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3043 2024-09-16T12:58:45.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:45.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:58:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:45.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:58:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:58:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:58:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:58:45.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:58:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:58:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:58:45.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:58:45.790 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T12:58:45.783+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8dd159860 2024-09-16T12:58:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:58:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:58:46.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:58:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3050 2024-09-16T12:58:46.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:46.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3049 2024-09-16T12:58:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3054 2024-09-16T12:58:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:58:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:58:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:58:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3055 2024-09-16T12:58:47.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:47.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:47.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3053 2024-09-16T12:58:47.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:48.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3058 2024-09-16T12:58:48.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:58:48.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:58:48.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:58:48.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:58:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:58:48.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:58:48.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:58:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:58:48.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:58:49.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3061 2024-09-16T12:58:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:58:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:58:49.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:58:49.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:58:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:58:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:58:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3063 2024-09-16T12:58:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3064 2024-09-16T12:58:50.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3065 2024-09-16T12:58:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:58:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:58:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:58:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:58:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:58:50.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:58:50.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:50.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:50.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:58:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:58:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:58:51.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:58:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:58:51.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:58:51.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3070 2024-09-16T12:58:51.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3069 2024-09-16T12:58:51.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:58:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:58:52.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:58:52.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3075 2024-09-16T12:58:52.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3072 2024-09-16T12:58:52.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:52.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3071 2024-09-16T12:58:52.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:52.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:53.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3076 2024-09-16T12:58:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:58:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:58:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:58:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:58:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:58:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:58:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:53.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:58:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:58:53.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:58:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3081 2024-09-16T12:58:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:58:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:54.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:58:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:58:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:58:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:58:54.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:58:54.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:55.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:58:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:58:55.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:58:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:58:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:58:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:58:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3084 2024-09-16T12:58:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:58:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:58:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:58:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3085 2024-09-16T12:58:55.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:55.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3083 2024-09-16T12:58:55.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:55.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:56.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:58:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:58:56.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:58:56.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3089 2024-09-16T12:58:56.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:56.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3090 2024-09-16T12:58:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:57.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3093 2024-09-16T12:58:57.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3091 2024-09-16T12:58:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:58:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:58:57.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:58:57.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:57.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:57.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3092 2024-09-16T12:58:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:58.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3096 2024-09-16T12:58:58.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:58:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:58:58.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:58:58.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:58:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:58:58.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:58:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:58:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:58:58.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:58:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:58:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:58:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:58:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3101 2024-09-16T12:58:59.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:58:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:58:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:58:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:58:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:58:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:58:59.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:59:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:59:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:59:00.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:59:00.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3103 2024-09-16T12:59:00.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:00.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:59:00.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:00.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:00.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:59:00.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:59:00.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:00.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:59:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:59:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:59:00.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3105 2024-09-16T12:59:00.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3104 2024-09-16T12:59:00.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:00.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:01.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:59:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:59:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:59:01.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3110 2024-09-16T12:59:01.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3109 2024-09-16T12:59:01.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:01.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:59:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:59:02.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:59:02.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3112 2024-09-16T12:59:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3114 2024-09-16T12:59:02.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3111 2024-09-16T12:59:02.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:02.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:02.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3116 2024-09-16T12:59:03.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:03.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:59:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:59:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:59:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:59:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:59:03.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:59:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:59:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:59:03.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:59:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3121 2024-09-16T12:59:04.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:59:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:59:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:59:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:59:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:59:04.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:59:05.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3123 2024-09-16T12:59:05.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3125 2024-09-16T12:59:05.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3124 2024-09-16T12:59:05.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:05.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:05.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:59:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:59:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:59:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:59:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:59:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:59:05.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:59:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:59:05.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:59:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:59:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:59:06.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:59:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3129 2024-09-16T12:59:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3130 2024-09-16T12:59:06.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:07.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:59:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:59:07.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:59:07.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3135 2024-09-16T12:59:07.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3134 2024-09-16T12:59:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:07.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:07.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3133 2024-09-16T12:59:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:59:08.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:59:08.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:59:08.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:59:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:59:08.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:59:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:59:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:59:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:59:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3136 2024-09-16T12:59:08.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3140 2024-09-16T12:59:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:59:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:09.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:59:09.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:59:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:59:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:59:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:59:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3142 2024-09-16T12:59:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:59:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:59:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:59:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:59:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:59:10.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:59:10.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:59:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:10.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:59:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:59:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3144 2024-09-16T12:59:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3143 2024-09-16T12:59:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:10.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:11.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3149 2024-09-16T12:59:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:59:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:59:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:59:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3150 2024-09-16T12:59:11.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:11.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:59:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:59:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:59:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3151 2024-09-16T12:59:12.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3153 2024-09-16T12:59:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3152 2024-09-16T12:59:12.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:12.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3159 2024-09-16T12:59:13.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:59:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:59:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:59:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:59:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:13.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:59:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:59:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:59:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:59:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:59:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3160 2024-09-16T12:59:14.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:14.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:59:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:59:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:59:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:59:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:14.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:59:14.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:59:15.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3164 2024-09-16T12:59:15.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3165 2024-09-16T12:59:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3163 2024-09-16T12:59:15.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:15.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:59:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:59:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:59:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:59:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:59:15.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:59:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:59:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:15.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:59:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:59:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3170 2024-09-16T12:59:16.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3169 2024-09-16T12:59:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:59:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:59:16.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:59:16.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:16.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:17.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3173 2024-09-16T12:59:17.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3174 2024-09-16T12:59:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:59:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:59:17.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:59:17.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3175 2024-09-16T12:59:17.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:17.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:17.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:18.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3179 2024-09-16T12:59:18.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:59:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:18.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:59:18.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:59:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:59:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:18.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:59:18.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:59:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:59:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:59:18.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:59:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3181 2024-09-16T12:59:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:59:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:59:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:59:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:59:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:59:19.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:59:20.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3185 2024-09-16T12:59:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3183 2024-09-16T12:59:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:59:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:59:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:59:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:59:20.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:59:20.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:59:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T12:59:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:20.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T12:59:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T12:59:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3184 2024-09-16T12:59:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:20.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:20.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:59:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:59:21.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:59:21.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3190 2024-09-16T12:59:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3189 2024-09-16T12:59:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:59:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:59:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:59:22.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3191 2024-09-16T12:59:22.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3192 2024-09-16T12:59:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3193 2024-09-16T12:59:22.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:22.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:22.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:23.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3196 2024-09-16T12:59:23.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:59:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:59:23.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:59:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:59:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:59:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:59:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T12:59:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T12:59:23.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T12:59:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:59:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:59:24.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:59:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3200 2024-09-16T12:59:24.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T12:59:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T12:59:24.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T12:59:25.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3203 2024-09-16T12:59:25.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3204 2024-09-16T12:59:25.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:59:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:59:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:59:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:59:25.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 74778 2024-09-16T12:59:25.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:25.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:59:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:59:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:59:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:25.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3205 2024-09-16T12:59:25.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:25.699 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:59:25.690+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e25b3520 2024-09-16T12:59:25.699 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:59:25.690+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e36aa000 2024-09-16T12:59:25.699 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:59:25.690+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e36ab1e0 2024-09-16T12:59:25.700 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:59:25.690+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e5d9d2b0 2024-09-16T12:59:25.701 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T12:59:25.694+0000 7f315ffe6700 -1 reset not still connected to 0x55c1f8dd4c30 2024-09-16T12:59:26.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:59:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:59:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:59:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3210 2024-09-16T12:59:26.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3209 2024-09-16T12:59:26.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T12:59:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T12:59:27.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T12:59:27.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3213 2024-09-16T12:59:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3212 2024-09-16T12:59:27.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:28.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3215 2024-09-16T12:59:28.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:59:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:59:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:59:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:59:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:59:28.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:59:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:59:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 74913 2024-09-16T12:59:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3219 2024-09-16T12:59:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:59:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 75084 2024-09-16T12:59:29.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T12:59:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T12:59:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T12:59:30.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3221 2024-09-16T12:59:30.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3222 2024-09-16T12:59:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:59:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:59:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:59:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:59:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:59:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:59:30.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:30.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T12:59:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T12:59:31.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T12:59:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3225 2024-09-16T12:59:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:59:32.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 75243 2024-09-16T12:59:32.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3227 2024-09-16T12:59:32.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3228 2024-09-16T12:59:32.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:32.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:33.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3231 2024-09-16T12:59:33.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T12:59:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T12:59:33.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T12:59:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:59:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:33.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:59:33.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:59:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:59:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 75458 2024-09-16T12:59:35.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3234 2024-09-16T12:59:35.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3233 2024-09-16T12:59:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:59:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:59:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:59:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T12:59:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T12:59:35.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T12:59:35.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:59:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 75628 2024-09-16T12:59:37.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3237 2024-09-16T12:59:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3238 2024-09-16T12:59:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:59:38.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 75831 2024-09-16T12:59:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T12:59:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T12:59:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T12:59:40.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3241 2024-09-16T12:59:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:59:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 76027 2024-09-16T12:59:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T12:59:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T12:59:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T12:59:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T12:59:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T12:59:40.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T12:59:42.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3243 2024-09-16T12:59:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T12:59:43.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T12:59:43.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 76325 2024-09-16T12:59:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T12:59:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T12:59:45.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 74644 2024-09-16T13:01:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:01:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:01:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:01:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 74779 2024-09-16T13:01:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:01:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:01:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:01:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 74914 2024-09-16T13:02:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:02:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:02:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:02:01.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 75085 2024-09-16T13:02:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:02:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:02:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:02:03.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 75244 2024-09-16T13:02:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:02:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:02:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:02:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 75459 2024-09-16T13:02:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:02:07.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:02:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:02:07.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 75631 2024-09-16T13:02:09.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:02:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:02:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:02:09.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 75832 2024-09-16T13:02:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:02:11.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:02:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:02:11.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 76028 2024-09-16T13:02:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:02:14.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:02:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:02:14.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 76326 2024-09-16T13:02:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:02:16.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:02:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T13:02:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:02:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:02:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:02:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 1 2024-09-16T13:02:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:02:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd0 2024-09-16T13:02:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary1 2024-09-16T13:02:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:02:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary1 2024-09-16T13:02:16.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:02:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary1 2024-09-16T13:02:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 2 2024-09-16T13:02:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:02:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd1 2024-09-16T13:02:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary2 2024-09-16T13:02:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:02:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary2 2024-09-16T13:02:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:02:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary2 2024-09-16T13:02:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 3 2024-09-16T13:02:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:02:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:02:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd2 2024-09-16T13:02:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 4 2024-09-16T13:02:16.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary3 2024-09-16T13:02:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary3 2024-09-16T13:02:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:02:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary3 2024-09-16T13:02:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:02:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd3 2024-09-16T13:02:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary4 2024-09-16T13:02:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:02:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary4 2024-09-16T13:02:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:02:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary4 2024-09-16T13:02:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 5 2024-09-16T13:02:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd4 2024-09-16T13:02:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:02:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary5 2024-09-16T13:02:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary5 2024-09-16T13:02:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:02:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary5 2024-09-16T13:02:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:02:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 6 2024-09-16T13:02:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd5 2024-09-16T13:02:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:02:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary6 2024-09-16T13:02:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary6 2024-09-16T13:02:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:02:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary6 2024-09-16T13:02:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:02:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 7 2024-09-16T13:02:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd6 2024-09-16T13:02:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:02:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary7 2024-09-16T13:02:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:02:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary7 2024-09-16T13:02:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:02:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary7 2024-09-16T13:02:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 8 2024-09-16T13:02:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:02:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd7 2024-09-16T13:02:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary8 2024-09-16T13:02:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary8 2024-09-16T13:02:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:02:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary8 2024-09-16T13:02:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 9 2024-09-16T13:02:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:02:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd8 2024-09-16T13:02:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary9 2024-09-16T13:02:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:02:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary9 2024-09-16T13:02:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:02:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary9 2024-09-16T13:02:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:02:16.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 82110 2024-09-16T13:02:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 10 2024-09-16T13:02:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd9 2024-09-16T13:02:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary10 2024-09-16T13:02:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary10 2024-09-16T13:02:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:02:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary10 2024-09-16T13:02:16.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd2 2024-09-16T13:02:16.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:02:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd5 2024-09-16T13:02:16.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:02:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:02:16.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd6 2024-09-16T13:02:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd0 2024-09-16T13:02:16.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:02:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd8 2024-09-16T13:02:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:02:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd7 2024-09-16T13:02:16.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:02:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:02:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd3 2024-09-16T13:02:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd9 2024-09-16T13:02:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:02:17.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:02:17.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd1 2024-09-16T13:02:17.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd4 2024-09-16T13:02:17.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:03:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=a8ca70bad3a4fcc0620cb7212d485403 2024-09-16T13:03:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary8 2024-09-16T13:03:12.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=9164ff6a19482f1c5a018164920a7837 2024-09-16T13:03:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary3 2024-09-16T13:03:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=d8a2a96fb53b3578fa9b36757ef52813 2024-09-16T13:03:13.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary6 2024-09-16T13:03:13.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=ce692e395be6a0350e1694bda4a6955d 2024-09-16T13:03:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary1 2024-09-16T13:03:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=e0726a558905fc781d10bed94ca34fe1 2024-09-16T13:03:14.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary2 2024-09-16T13:03:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=a7b5ed4fd38e80ccc5a5ea2b1c922450 2024-09-16T13:03:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary4 2024-09-16T13:03:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=22a77670629a42823d5e3291452cbfe8 2024-09-16T13:03:14.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary7 2024-09-16T13:03:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-16T13:03:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:03:14.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-16T13:03:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-16T13:03:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:03:14.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-16T13:03:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=fe459229c56816eeee4cd0d9423587a2 2024-09-16T13:03:14.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary10 2024-09-16T13:03:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=b5dc4eed05d4e4665a471eed085192a6 2024-09-16T13:03:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary5 2024-09-16T13:03:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=72f0bd54aed513764884e39da0c8f975 2024-09-16T13:03:14.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary9 2024-09-16T13:03:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-16T13:03:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:03:14.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-16T13:03:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-16T13:03:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:03:14.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-16T13:03:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-16T13:03:14.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:14.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:14.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:03:14.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-16T13:03:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-16T13:03:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:14.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:03:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-16T13:03:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-16T13:03:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:03:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-16T13:03:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-16T13:03:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:15.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:03:15.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-16T13:03:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-16T13:03:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:03:15.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-16T13:03:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-16T13:03:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:03:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-16T13:03:16.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:03:16.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:03:16.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:03:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-16T13:03:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:03:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:16.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:03:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:03:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:03:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-16T13:03:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:03:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:03:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:03:16.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:03:16.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-16T13:03:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:03:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:16.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:03:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-16T13:03:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:03:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:03:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-16T13:03:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:16.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:03:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:16.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:03:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-16T13:03:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:03:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:16.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:03:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-16T13:03:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:03:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:16.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:03:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:03:17.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:03:17.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:03:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-16T13:03:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:03:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:17.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:17.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:03:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-16T13:03:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:03:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:17.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:03:17.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-16T13:03:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:03:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:17.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:03:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-16T13:03:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-16T13:03:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:17.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:03:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:03:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-16T13:03:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:17.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:03:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-16T13:03:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:03:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:17.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-16T13:03:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:03:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T13:03:17.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-16T13:03:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-16T13:03:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:03:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:03:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:17.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T13:03:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T13:03:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T13:03:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:17.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T13:03:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T13:03:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:17.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T13:03:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:03:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a9797a82-534a-4a13-8444-a14e2d2586a3 2024-09-16T13:03:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:03:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:03:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:13 2024-09-16T13:03:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:03:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:03:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1705984.0,"last_snapshot_bytes":1597440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491561,"remote_snapshot_timestamp":1726491561,"replay_state":"idle"} 2024-09-16T13:03:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:02:53' 2024-09-16T13:03:17.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:03:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b18ebe6c-4efe-4743-8aee-94780a9c5a06 2024-09-16T13:03:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:03:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:03:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:13 2024-09-16T13:03:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:03:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:03:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2144256.0,"last_snapshot_bytes":2588672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491572,"remote_snapshot_timestamp":1726491572,"replay_state":"idle"} 2024-09-16T13:03:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:02:53' 2024-09-16T13:03:17.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:03:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 90dd28e8-9a26-4b71-8b37-c0cde9f6ea0e 2024-09-16T13:03:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:03:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:03:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:13 2024-09-16T13:03:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:03:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:03:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3192832.0,"last_snapshot_bytes":2617344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491568,"remote_snapshot_timestamp":1726491568,"replay_state":"idle"} 2024-09-16T13:03:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:02:53' 2024-09-16T13:03:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:17.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:03:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: f491577d-20f7-4b88-9fb4-e72a02bb468c 2024-09-16T13:03:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:03:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:03:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:13 2024-09-16T13:03:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:03:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:03:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2150400.0,"last_snapshot_bytes":2555904,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491574,"remote_snapshot_timestamp":1726491574,"replay_state":"idle"} 2024-09-16T13:03:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:02:53' 2024-09-16T13:03:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:03:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: c7162764-8ecd-41dd-bf00-31141188bb23 2024-09-16T13:03:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:03:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:03:17.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:13 2024-09-16T13:03:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:03:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:03:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2140160.0,"last_snapshot_bytes":2609152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491576,"remote_snapshot_timestamp":1726491576,"replay_state":"idle"} 2024-09-16T13:03:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:02:53' 2024-09-16T13:03:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:17.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:03:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 196c49a1-a2e7-49aa-a4ea-cf8227ec3d71 2024-09-16T13:03:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:03:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:03:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:13 2024-09-16T13:03:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:03:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:03:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1693696.0,"last_snapshot_bytes":1691648,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491564,"remote_snapshot_timestamp":1726491564,"replay_state":"idle"} 2024-09-16T13:03:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:02:53' 2024-09-16T13:03:17.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:03:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ff 2024-09-16T13:03:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:03:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:03:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:13 2024-09-16T13:03:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:03:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:03:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3813376.0,"last_snapshot_bytes":3825664,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491565,"remote_snapshot_timestamp":1726491565,"replay_state":"idle"} 2024-09-16T13:03:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:02:53' 2024-09-16T13:03:17.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:03:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:03:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:03:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:03:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:03:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:17.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:03:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:03:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:03:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:03:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:03:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:03:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:03:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:03:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:03:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-16T13:03:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:18.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:03:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:18.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T13:03:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-16T13:03:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:03:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:18.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:18.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:18.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T13:03:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-16T13:03:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:03:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:18.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T13:03:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:03:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 18c6fc24-69f8-4190-a25c-6b81712dda2c 2024-09-16T13:03:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:03:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:03:18.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:13 2024-09-16T13:03:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:03:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:03:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:17' 2024-09-16T13:03:18.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:03:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:18.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:03:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:03:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7 2024-09-16T13:03:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:03:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:03:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:13 2024-09-16T13:03:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:18.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:03:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:03:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:17' 2024-09-16T13:03:18.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:03:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:18.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:03:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:03:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50 2024-09-16T13:03:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:03:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:03:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:13 2024-09-16T13:03:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:03:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:03:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:17' 2024-09-16T13:03:18.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:03:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:03:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-16T13:03:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-16T13:03:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-16T13:03:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:03:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-16T13:03:19.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:19.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:19.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:19.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:03:19.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:19.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:19.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:19.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:19.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:19.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:19.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:03:19.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:19.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:19.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:19.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:03:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-16T13:03:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-16T13:03:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:03:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-16T13:03:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:03:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:03:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T13:03:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T13:03:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T13:03:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T13:03:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T13:03:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T13:03:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T13:03:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:03:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b18ebe6c-4efe-4743-8aee-94780a9c5a06 2024-09-16T13:03:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18 2024-09-16T13:03:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:03:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:19.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18' 2024-09-16T13:03:19.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:03:19.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:03:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:03:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:03:19.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:03:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:03:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ff 2024-09-16T13:03:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:19.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18 2024-09-16T13:03:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:03:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18' 2024-09-16T13:03:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:19.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:03:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:03:19.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:03:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:03:19.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:03:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:03:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:03:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:19.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:03:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:03:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:03:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 90dd28e8-9a26-4b71-8b37-c0cde9f6ea0e 2024-09-16T13:03:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18 2024-09-16T13:03:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:03:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:19.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18' 2024-09-16T13:03:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:19.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:03:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:03:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:03:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:03:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a9797a82-534a-4a13-8444-a14e2d2586a3 2024-09-16T13:03:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18 2024-09-16T13:03:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:19.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:03:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18' 2024-09-16T13:03:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:03:19.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:03:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:03:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:03:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:03:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:03:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:03:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:03:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:03:19.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: c7162764-8ecd-41dd-bf00-31141188bb23 2024-09-16T13:03:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18 2024-09-16T13:03:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:03:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18' 2024-09-16T13:03:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:03:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:03:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:03:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 196c49a1-a2e7-49aa-a4ea-cf8227ec3d71 2024-09-16T13:03:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18 2024-09-16T13:03:19.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:03:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18' 2024-09-16T13:03:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:03:19.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:03:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:03:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:03:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:03:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:03:19.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:03:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:03:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:03:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:03:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:03:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:03:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:03:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:03:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: f491577d-20f7-4b88-9fb4-e72a02bb468c 2024-09-16T13:03:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18 2024-09-16T13:03:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:03:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18' 2024-09-16T13:03:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:03:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:03:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:03:19.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:03:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:03:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:03:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:03:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:03:19.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:03:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:03:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:03:19.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:03:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:03:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:03:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:03:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:03:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:03:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:03:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:03:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:03:19.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:03:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:03:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:03:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-16 13:03:18' 2024-09-16T13:03:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:03:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:19.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:03:18' 2024-09-16T13:03:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-16 13:03:18' 2024-09-16T13:03:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-16 13:03:18' 2024-09-16T13:03:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:03:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:03:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:19.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:03:18' 2024-09-16T13:03:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:03:18' 2024-09-16T13:03:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-16 13:03:18' 2024-09-16T13:03:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-16 13:03:18' 2024-09-16T13:03:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:03:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:03:18' 2024-09-16T13:03:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:19.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:03:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:03:18' 2024-09-16T13:03:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-16 13:03:18' 2024-09-16T13:03:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:03:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:03:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:03:18' 2024-09-16T13:03:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-16 13:03:18' 2024-09-16T13:03:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:03:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:03:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:03:18' 2024-09-16T13:03:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:03:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:03:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:03:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:03:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:19.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:03:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18' 2024-09-16T13:03:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:03:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T13:03:20.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18' 2024-09-16T13:03:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:03:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T13:03:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18' 2024-09-16T13:03:20.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:03:20.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T13:03:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18' 2024-09-16T13:03:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:03:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T13:03:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T13:03:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T13:03:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18' 2024-09-16T13:03:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:03:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T13:03:20.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T13:03:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:03:20.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:03:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T13:03:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:03:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:03:20.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:03:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:03:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18' 2024-09-16T13:03:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:03:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T13:03:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18' 2024-09-16T13:03:20.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:03:20.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T13:03:20.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:20.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:20.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:20.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-16T13:03:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:03:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-16T13:03:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T13:03:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:03:20.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:03:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:03:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-16T13:03:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:03:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:20.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:03:20.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:03:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:03:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:03:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:03:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:20.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T13:03:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T13:03:20.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-16T13:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:20.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:03:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-16T13:03:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:03:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:03:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:03:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:03:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:03:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:03:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:20.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:20.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-16T13:03:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:03:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-16T13:03:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:03:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:03:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:20.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:03:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-16T13:03:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:20.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:03:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:20.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T13:03:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-16T13:03:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:03:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:20.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T13:03:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-16T13:03:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:03:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T13:03:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:03:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7 2024-09-16T13:03:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18 2024-09-16T13:03:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:03:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18' 2024-09-16T13:03:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:03:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:03:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:03:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 18c6fc24-69f8-4190-a25c-6b81712dda2c 2024-09-16T13:03:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18 2024-09-16T13:03:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:03:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18' 2024-09-16T13:03:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:03:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:03:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:03:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:03:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:03:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:03:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:03:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:03:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:03:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:03:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:03:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:03:20.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:03:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:03:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:03:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:03:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-16 13:03:18' 2024-09-16T13:03:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:03:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:03:18' 2024-09-16T13:03:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:20.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:20.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:03:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-16 13:03:18' 2024-09-16T13:03:20.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:03:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:03:18' 2024-09-16T13:03:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:03:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:03:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50 2024-09-16T13:03:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18 2024-09-16T13:03:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:03:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18' 2024-09-16T13:03:20.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:03:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:03:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:03:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:03:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:20.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:03:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:03:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:03:20.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:03:20.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:03:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:20.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-16 13:03:18' 2024-09-16T13:03:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:03:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:03:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:03:18' 2024-09-16T13:03:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:03:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18' 2024-09-16T13:03:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:03:20.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T13:03:20.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:20.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T13:03:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:03:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:03:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-16T13:03:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:03:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:20.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:03:20.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18' 2024-09-16T13:03:20.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:03:20.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T13:03:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:20.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T13:03:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:03:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:03:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-16T13:03:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:03:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:03:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:03:18cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:03:18' 2024-09-16T13:03:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:03:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T13:03:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:20.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T13:03:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:03:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:20.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:03:20.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-16T13:03:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:20.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:03:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:03:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:03:21.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-16T13:03:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:03:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-16T13:03:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:03:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:21.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-16T13:03:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:03:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-16T13:03:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-16T13:03:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:21.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-16T13:03:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-16T13:03:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:03:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-16T13:03:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:03:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-16T13:03:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:21.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-16T13:03:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-16T13:03:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-16T13:03:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:03:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:03:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:21.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-16T13:03:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-16T13:03:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:03:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ff 2024-09-16T13:03:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18 2024-09-16T13:03:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:03:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18' 2024-09-16T13:03:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:03:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:03:21.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:03:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:03:21.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:21.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:03:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:21.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:03:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:03:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:03:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:03:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:03:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a9797a82-534a-4a13-8444-a14e2d2586a3 2024-09-16T13:03:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18 2024-09-16T13:03:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:03:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18' 2024-09-16T13:03:21.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:21.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:03:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 196c49a1-a2e7-49aa-a4ea-cf8227ec3d71 2024-09-16T13:03:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:21.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18 2024-09-16T13:03:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:03:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18' 2024-09-16T13:03:21.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-16 13:03:18' 2024-09-16T13:03:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:03:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:21.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:03:18' 2024-09-16T13:03:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:03:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:03:21.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:03:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:03:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:03:21.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:03:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:03:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:03:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:03:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 90dd28e8-9a26-4b71-8b37-c0cde9f6ea0e 2024-09-16T13:03:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18 2024-09-16T13:03:21.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:03:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18' 2024-09-16T13:03:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:03:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:03:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: f491577d-20f7-4b88-9fb4-e72a02bb468c 2024-09-16T13:03:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18 2024-09-16T13:03:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:03:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18' 2024-09-16T13:03:21.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:03:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: c7162764-8ecd-41dd-bf00-31141188bb23 2024-09-16T13:03:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18 2024-09-16T13:03:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:03:21.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18' 2024-09-16T13:03:21.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:03:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:03:21.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:03:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:03:21.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:03:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:03:21.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:03:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:03:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:03:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b18ebe6c-4efe-4743-8aee-94780a9c5a06 2024-09-16T13:03:21.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18 2024-09-16T13:03:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:03:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18' 2024-09-16T13:03:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:03:21.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:03:21.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:03:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:03:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:03:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:03:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:03:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:03:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:03:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-16 13:03:18' 2024-09-16T13:03:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:03:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:03:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:03:18' 2024-09-16T13:03:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:03:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:03:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:03:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:03:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:03:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:03:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:03:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:03:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:03:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:03:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:03:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:03:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:03:21.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:03:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:03:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:03:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:03:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:03:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:03:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:03:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-16 13:03:18' 2024-09-16T13:03:21.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:03:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:03:18' 2024-09-16T13:03:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:03:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:03:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:03:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-16 13:03:18' 2024-09-16T13:03:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:03:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:03:18' 2024-09-16T13:03:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:03:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-16 13:03:18' 2024-09-16T13:03:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:03:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:03:18' 2024-09-16T13:03:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:03:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:03:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:03:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:03:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:03:21.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:03:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:03:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:03:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:21.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-16 13:03:18' 2024-09-16T13:03:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:03:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:03:18' 2024-09-16T13:03:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:03:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-16 13:03:18' 2024-09-16T13:03:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:03:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:03:18' 2024-09-16T13:03:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:03:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18' 2024-09-16T13:03:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:03:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T13:03:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:21.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T13:03:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:03:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:03:21.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-16T13:03:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:03:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:03:21.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-16T13:03:21.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18' 2024-09-16T13:03:21.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:03:21.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T13:03:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:21.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T13:03:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:03:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:03:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:21.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-16T13:03:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:03:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:03:21.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-16T13:03:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18' 2024-09-16T13:03:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:03:21.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T13:03:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:21.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T13:03:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:03:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18' 2024-09-16T13:03:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:03:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:03:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T13:03:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:21.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-16T13:03:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:03:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:03:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-16T13:03:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T13:03:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18' 2024-09-16T13:03:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:03:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18' 2024-09-16T13:03:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:03:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T13:03:21.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T13:03:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18' 2024-09-16T13:03:21.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:03:21.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:03:21.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:21.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:03:21.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:21.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:21.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:21.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-16T13:03:21.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:03:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:03:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-16T13:03:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T13:03:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T13:03:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T13:03:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:03:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:03:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:21.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T13:03:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:03:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:21.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-16T13:03:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:03:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:03:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-16T13:03:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:03:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:21.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-16T13:03:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:03:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:03:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-16T13:03:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:03:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:21.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:03:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-16T13:03:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:03:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:03:21.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-16T13:03:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-16T13:03:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:03:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:21.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:21.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-16T13:03:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-16T13:03:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:03:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-16T13:03:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:03:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:21.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:03:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:03:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-16T13:03:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:21.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-16T13:03:21.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:03:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7 2024-09-16T13:03:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18 2024-09-16T13:03:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:03:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18' 2024-09-16T13:03:21.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:03:21.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:03:21.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:03:21.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:03:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:03:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:03:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:03:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:03:21.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:03:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-16 13:03:18' 2024-09-16T13:03:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:03:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:03:18' 2024-09-16T13:03:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:21.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:03:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:03:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50 2024-09-16T13:03:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18 2024-09-16T13:03:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:03:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:21.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18' 2024-09-16T13:03:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:03:21.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:03:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:03:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:03:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:03:21.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:21.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:03:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:03:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:03:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:03:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-16 13:03:18' 2024-09-16T13:03:21.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:03:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:03:18' 2024-09-16T13:03:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:03:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18' 2024-09-16T13:03:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:03:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T13:03:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:21.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T13:03:21.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:03:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:03:21.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-16T13:03:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:03:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:03:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-16T13:03:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18' 2024-09-16T13:03:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:03:21.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T13:03:21.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:21.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T13:03:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:03:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:03:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:21.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-16T13:03:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:03:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:03:21.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-16T13:03:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:03:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:03:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:03:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:03:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:03:21.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:03:21.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:03:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:03:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary4 2024-09-16T13:03:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:03:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:03:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary1 2024-09-16T13:03:21.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary8 2024-09-16T13:03:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:03:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary2 2024-09-16T13:03:21.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:03:21.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 18c6fc24-69f8-4190-a25c-6b81712dda2c 2024-09-16T13:03:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:03:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18 2024-09-16T13:03:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:03:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:03:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:03:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:03:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:03:18' 2024-09-16T13:03:21.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:03:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:03:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary3 2024-09-16T13:03:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:03:21.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:03:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:03:21.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:03:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:03:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:03:21.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary6 2024-09-16T13:03:21.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary7 2024-09-16T13:03:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:03:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:21.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:03:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:03:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:03:21.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:03:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-16 13:03:18' 2024-09-16T13:03:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:03:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:03:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:03:18' 2024-09-16T13:03:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:03:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:03:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:03:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:03:18cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:03:18' 2024-09-16T13:03:22.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:03:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T13:03:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:03:18' 2024-09-16T13:03:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T13:03:22.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd0 2024-09-16T13:03:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:03:22.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd0 2024-09-16T13:03:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:03:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:03:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:03:22.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:03:22.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:22.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:03:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-16T13:03:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:03:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:03:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:03:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:03:22.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-16T13:03:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd1 2024-09-16T13:03:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd1 2024-09-16T13:03:22.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:03:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd2 2024-09-16T13:03:22.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd2 2024-09-16T13:03:22.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:03:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd3 2024-09-16T13:03:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:03:22.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd3 2024-09-16T13:03:22.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd4 2024-09-16T13:03:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:03:22.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd4 2024-09-16T13:03:22.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd5 2024-09-16T13:03:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:03:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd5 2024-09-16T13:03:22.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd6 2024-09-16T13:03:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:03:22.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd6 2024-09-16T13:03:22.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:03:22.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:03:22.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:03:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary10 2024-09-16T13:03:22.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:03:22.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:03:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary5 2024-09-16T13:03:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:03:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary9 2024-09-16T13:03:23.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd7 2024-09-16T13:03:23.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:03:23.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd7 2024-09-16T13:03:23.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd8 2024-09-16T13:03:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:03:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd8 2024-09-16T13:03:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd9 2024-09-16T13:03:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:03:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd9 2024-09-16T13:04:56.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=a7b5ed4fd38e80ccc5a5ea2b1c922450 2024-09-16T13:04:56.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd1 2024-09-16T13:04:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=9164ff6a19482f1c5a018164920a7837 2024-09-16T13:04:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd5 2024-09-16T13:04:58.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ a7b5ed4fd38e80ccc5a5ea2b1c922450 != \a\7\b\5\e\d\4\f\d\3\8\e\8\0\c\c\c\5\a\5\e\a\2\b\1\c\9\2\2\4\5\0 ]] 2024-09-16T13:04:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=e0726a558905fc781d10bed94ca34fe1 2024-09-16T13:04:58.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd3 2024-09-16T13:04:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=d8a2a96fb53b3578fa9b36757ef52813 2024-09-16T13:04:58.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd6 2024-09-16T13:04:59.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 9164ff6a19482f1c5a018164920a7837 != \9\1\6\4\f\f\6\a\1\9\4\8\2\f\1\c\5\a\0\1\8\1\6\4\9\2\0\a\7\8\3\7 ]] 2024-09-16T13:04:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=72f0bd54aed513764884e39da0c8f975 2024-09-16T13:04:59.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd9 2024-09-16T13:04:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=a8ca70bad3a4fcc0620cb7212d485403 2024-09-16T13:04:59.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd2 2024-09-16T13:04:59.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ e0726a558905fc781d10bed94ca34fe1 != \e\0\7\2\6\a\5\5\8\9\0\5\f\c\7\8\1\d\1\0\b\e\d\9\4\c\a\3\4\f\e\1 ]] 2024-09-16T13:04:59.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ d8a2a96fb53b3578fa9b36757ef52813 != \d\8\a\2\a\9\6\f\b\5\3\b\3\5\7\8\f\a\9\b\3\6\7\5\7\e\f\5\2\8\1\3 ]] 2024-09-16T13:04:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=22a77670629a42823d5e3291452cbfe8 2024-09-16T13:04:59.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd4 2024-09-16T13:04:59.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=ce692e395be6a0350e1694bda4a6955d 2024-09-16T13:04:59.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-09-16T13:05:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=fe459229c56816eeee4cd0d9423587a2 2024-09-16T13:05:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd7 2024-09-16T13:05:00.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 72f0bd54aed513764884e39da0c8f975 != \7\2\f\0\b\d\5\4\a\e\d\5\1\3\7\6\4\8\8\4\e\3\9\d\a\0\c\8\f\9\7\5 ]] 2024-09-16T13:05:00.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ a8ca70bad3a4fcc0620cb7212d485403 != \a\8\c\a\7\0\b\a\d\3\a\4\f\c\c\0\6\2\0\c\b\7\2\1\2\d\4\8\5\4\0\3 ]] 2024-09-16T13:05:00.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 22a77670629a42823d5e3291452cbfe8 != \2\2\a\7\7\6\7\0\6\2\9\a\4\2\8\2\3\d\5\e\3\2\9\1\4\5\2\c\b\f\e\8 ]] 2024-09-16T13:05:00.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=b5dc4eed05d4e4665a471eed085192a6 2024-09-16T13:05:00.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd8 2024-09-16T13:05:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ ce692e395be6a0350e1694bda4a6955d != \c\e\6\9\2\e\3\9\5\b\e\6\a\0\3\5\0\e\1\6\9\4\b\d\a\4\a\6\9\5\5\d ]] 2024-09-16T13:05:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:05:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 82111 2024-09-16T13:05:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:05:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 82112 2024-09-16T13:05:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:05:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 82114 2024-09-16T13:05:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:05:00.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 82116 2024-09-16T13:05:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ fe459229c56816eeee4cd0d9423587a2 != \f\e\4\5\9\2\2\9\c\5\6\8\1\6\e\e\e\e\4\c\d\0\d\9\4\2\3\5\8\7\a\2 ]] 2024-09-16T13:05:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ b5dc4eed05d4e4665a471eed085192a6 != \b\5\d\c\4\e\e\d\0\5\d\4\e\4\6\6\5\a\4\7\1\e\e\d\0\8\5\1\9\2\a\6 ]] 2024-09-16T13:05:00.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:05:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 82118 2024-09-16T13:05:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:05:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 82121 2024-09-16T13:05:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:05:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 82123 2024-09-16T13:05:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:05:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 82124 2024-09-16T13:05:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:05:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 82127 2024-09-16T13:05:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T13:05:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:05:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary1 2024-09-16T13:05:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-16T13:05:00.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:05:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:05:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:05:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:05:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-16T13:05:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:05:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:05:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:05:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:05:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:05:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T13:05:01.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:05:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a9797a82-534a-4a13-8444-a14e2d2586a3 2024-09-16T13:05:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:05:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72159232.0,"last_snapshot_bytes":72159232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"} 2024-09-16T13:05:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:05:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:04:53 2024-09-16T13:05:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:05:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:05:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:05:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:05:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:04:58' 2024-09-16T13:05:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:05:01.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:05:01.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:05:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72159232.0,"last_snapshot_bytes":72159232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"} 2024-09-16T13:05:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:05:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:05:01.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:05:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:05:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:05:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:05:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:05:01.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:05:01.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:04:53' 2024-09-16T13:05:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-16 13:04:53' 2024-09-16T13:05:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:05:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:04:53' 2024-09-16T13:05:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:05:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:05:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:05:02.007 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T13:05:01.996+0000 7f4071c2f700 -1 reset not still connected to 0x5635bdf54820 2024-09-16T13:05:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a9797a82-534a-4a13-8444-a14e2d2586a3up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72159232.0,"last_snapshot_bytes":72159232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:04:58image-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70164480.0,"last_snapshot_bytes":70164480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:04:58image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70889472.0,"last_snapshot_bytes":70889472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:04:58image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":135204864.0,"last_snapshot_bytes":135204864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:04:58image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":106364928.0,"last_snapshot_bytes":106364928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:04:58image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70066176.0,"last_snapshot_bytes":70066176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:04:58image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70270976.0,"last_snapshot_bytes":70270976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:04:58image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70205440.0,"last_snapshot_bytes":70205440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:04:58image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70258688.0,"last_snapshot_bytes":70258688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:04:58image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70127616.0,"last_snapshot_bytes":70127616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:04:58' 2024-09-16T13:05:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:05:02.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T13:05:02.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:04:53' 2024-09-16T13:05:02.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T13:05:02.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:05:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:05:02.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:05:02.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:05:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-16T13:05:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:05:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:05:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-16T13:05:02.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-16T13:05:04.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:05:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-16T13:05:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:05:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:04.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary1 2024-09-16T13:05:04.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:05:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:04.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary1 2024-09-16T13:05:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:05:04.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary2 2024-09-16T13:05:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-16T13:05:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:05:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:05:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:05:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:04.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:05:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-16T13:05:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:05.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:05:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:05:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:05:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:05:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:05:05.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T13:05:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:05:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 196c49a1-a2e7-49aa-a4ea-cf8227ec3d71 2024-09-16T13:05:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:05:05.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70889472.0,"last_snapshot_bytes":70889472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"} 2024-09-16T13:05:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:05:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:04:53 2024-09-16T13:05:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:05:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:05:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:05:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:05:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:05:03' 2024-09-16T13:05:05.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:05:05.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:05:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:05:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70889472.0,"last_snapshot_bytes":70889472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"} 2024-09-16T13:05:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:05:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:05:05.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:05:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:05:05.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:05:05.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:05:05.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:05:05.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:05:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:04:53' 2024-09-16T13:05:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-16 13:04:53' 2024-09-16T13:05:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:05.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:05:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:04:53' 2024-09-16T13:05:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:05:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:05:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:05:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK942544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70164480.0,"last_snapshot_bytes":70164480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:03image-primary2196c49a1-a2e7-49aa-a4ea-cf8227ec3d71up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70889472.0,"last_snapshot_bytes":70889472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:03image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":135204864.0,"last_snapshot_bytes":135204864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:03image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":106364928.0,"last_snapshot_bytes":106364928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:03image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70066176.0,"last_snapshot_bytes":70066176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:03image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70270976.0,"last_snapshot_bytes":70270976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:03image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70205440.0,"last_snapshot_bytes":70205440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:03image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70258688.0,"last_snapshot_bytes":70258688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:03image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70127616.0,"last_snapshot_bytes":70127616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:03' 2024-09-16T13:05:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:05:05.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T13:05:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:04:53' 2024-09-16T13:05:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T13:05:05.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:05:05.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:05:05.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:05:05.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:05:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-16T13:05:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:05:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:05:05.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-16T13:05:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-16T13:05:06.667 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:06.656+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174d00 2024-09-16T13:05:06.667 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:06.656+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T13:05:06.668 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:06.656+0000 7ff4f6946700 -1 reset not still connected to 0x557562000a90 2024-09-16T13:05:06.668 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:06.656+0000 7ff4f6946700 -1 reset not still connected to 0x557568810750 2024-09-16T13:05:06.668 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:06.656+0000 7ff4f6946700 -1 reset not still connected to 0x557578a2b5f0 2024-09-16T13:05:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:05:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-16T13:05:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:05:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:05:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary2 2024-09-16T13:05:08.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary2 2024-09-16T13:05:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:05:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary3 2024-09-16T13:05:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-16T13:05:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:05:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:05:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:05:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:05:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-16T13:05:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:05:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:05:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:05:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:05:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:05:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T13:05:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:05:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ff 2024-09-16T13:05:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:05:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":135204864.0,"last_snapshot_bytes":135204864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"} 2024-09-16T13:05:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:05:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:04:53 2024-09-16T13:05:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:05:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:05:09.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:05:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:05:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:05:08' 2024-09-16T13:05:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:05:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:05:09.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:05:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":135204864.0,"last_snapshot_bytes":135204864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"} 2024-09-16T13:05:09.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:05:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:05:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:05:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:05:09.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:05:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:05:09.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:05:09.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:05:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:04:53' 2024-09-16T13:05:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-16 13:04:53' 2024-09-16T13:05:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:05:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:04:53' 2024-09-16T13:05:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:05:09.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:05:09.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:05:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK842544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70164480.0,"last_snapshot_bytes":70164480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:08image-primary3cdc95e40-bfeb-4142-8b4b-5c7b01c7d2ffup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":135204864.0,"last_snapshot_bytes":135204864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:08image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":106364928.0,"last_snapshot_bytes":106364928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:08image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70066176.0,"last_snapshot_bytes":70066176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:08image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70270976.0,"last_snapshot_bytes":70270976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:08image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70205440.0,"last_snapshot_bytes":70205440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:08image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70258688.0,"last_snapshot_bytes":70258688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:08image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70127616.0,"last_snapshot_bytes":70127616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:08' 2024-09-16T13:05:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:05:09.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T13:05:09.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:04:53' 2024-09-16T13:05:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T13:05:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:05:09.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:05:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:05:09.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:05:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-16T13:05:09.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:05:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:05:09.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-16T13:05:09.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-16T13:05:10.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:05:10.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-16T13:05:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:05:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:05:10.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary3 2024-09-16T13:05:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary3 2024-09-16T13:05:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:05:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary4 2024-09-16T13:05:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-16T13:05:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:05:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:05:10.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:05:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:10.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:05:10.974 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:10.964+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1748f0 2024-09-16T13:05:10.979 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:10.972+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174d00 2024-09-16T13:05:10.980 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:10.972+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T13:05:10.983 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:10.972+0000 7ff4f6946700 -1 reset not still connected to 0x557562000a90 2024-09-16T13:05:10.983 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:10.972+0000 7ff4f6946700 -1 reset not still connected to 0x557562001110 2024-09-16T13:05:10.983 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:10.972+0000 7ff4f6946700 -1 reset not still connected to 0x5575637ebd40 2024-09-16T13:05:10.983 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:10.972+0000 7ff4f6946700 -1 reset not still connected to 0x557568810750 2024-09-16T13:05:11.960 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:05:11.948+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T13:05:11.960 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:05:11.948+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T13:05:11.960 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:05:11.948+0000 7f61b1855700 -1 reset not still connected to 0x557b50869ee0 2024-09-16T13:05:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-16T13:05:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:05:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:05:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:05:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:05:11.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:05:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T13:05:12.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:05:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 90dd28e8-9a26-4b71-8b37-c0cde9f6ea0e 2024-09-16T13:05:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:05:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":106364928.0,"last_snapshot_bytes":106364928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"} 2024-09-16T13:05:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:05:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:04:53 2024-09-16T13:05:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:05:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:05:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:05:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:05:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:05:08' 2024-09-16T13:05:12.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:05:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:05:12.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:05:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":106364928.0,"last_snapshot_bytes":106364928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"} 2024-09-16T13:05:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:05:12.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:05:12.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:05:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:05:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:05:12.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:05:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:05:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:05:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:04:53' 2024-09-16T13:05:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-16 13:04:53' 2024-09-16T13:05:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:05:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:04:53' 2024-09-16T13:05:12.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:05:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:05:12.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:05:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK742544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70164480.0,"last_snapshot_bytes":70164480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:08image-primary490dd28e8-9a26-4b71-8b37-c0cde9f6ea0eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":106364928.0,"last_snapshot_bytes":106364928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:08image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70066176.0,"last_snapshot_bytes":70066176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:08image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70270976.0,"last_snapshot_bytes":70270976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:08image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70205440.0,"last_snapshot_bytes":70205440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:08image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70258688.0,"last_snapshot_bytes":70258688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:08image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70127616.0,"last_snapshot_bytes":70127616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:08' 2024-09-16T13:05:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:05:12.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T13:05:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:04:53' 2024-09-16T13:05:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T13:05:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:05:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:05:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:05:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:05:12.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:12.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:12.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-16T13:05:12.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:05:12.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:12.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:05:12.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-16T13:05:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-16T13:05:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:05:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-16T13:05:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:05:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:13.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:05:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary4 2024-09-16T13:05:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:13.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary4 2024-09-16T13:05:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:05:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary5 2024-09-16T13:05:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-16T13:05:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:05:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:05:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:05:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:13.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:05:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-16T13:05:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:05:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:05:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:05:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:05:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:05:14.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T13:05:15.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:05:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50 2024-09-16T13:05:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:05:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70066176.0,"last_snapshot_bytes":70066176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"} 2024-09-16T13:05:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:05:15.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:04:53 2024-09-16T13:05:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:05:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:05:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:05:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:05:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:05:13' 2024-09-16T13:05:15.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:05:15.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:05:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:05:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70066176.0,"last_snapshot_bytes":70066176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"} 2024-09-16T13:05:15.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:05:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:05:15.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:05:15.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:05:15.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:05:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:05:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:05:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:05:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:04:53' 2024-09-16T13:05:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-16 13:04:53' 2024-09-16T13:05:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:15.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:05:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:04:53' 2024-09-16T13:05:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:05:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:05:15.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:05:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK642544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70164480.0,"last_snapshot_bytes":70164480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:13image-primary5e3e9ed09-1d0d-4414-a1e5-ebd76b8b9d50up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70066176.0,"last_snapshot_bytes":70066176,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:13image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70270976.0,"last_snapshot_bytes":70270976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:13image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70205440.0,"last_snapshot_bytes":70205440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:13image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70258688.0,"last_snapshot_bytes":70258688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:13image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70127616.0,"last_snapshot_bytes":70127616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:13' 2024-09-16T13:05:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:05:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T13:05:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:04:53' 2024-09-16T13:05:15.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T13:05:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:05:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:05:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:05:15.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:05:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:15.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-16T13:05:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:05:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:05:15.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-16T13:05:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-16T13:05:15.998 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T13:05:15.988+0000 7f4071c2f700 -1 reset not still connected to 0x563591d7d040 2024-09-16T13:05:15.999 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T13:05:15.988+0000 7f4071c2f700 -1 reset not still connected to 0x56359653e750 2024-09-16T13:05:16.005 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T13:05:15.992+0000 7f4071c2f700 -1 reset not still connected to 0x56359fa0d6c0 2024-09-16T13:05:16.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:05:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-16T13:05:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:05:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:05:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary5 2024-09-16T13:05:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary5 2024-09-16T13:05:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:05:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary6 2024-09-16T13:05:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-16T13:05:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:05:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:05:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:05:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:05:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-16T13:05:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:05:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:05:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:05:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:05:18.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:05:18.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T13:05:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:05:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b18ebe6c-4efe-4743-8aee-94780a9c5a06 2024-09-16T13:05:18.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:05:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70270976.0,"last_snapshot_bytes":70270976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"} 2024-09-16T13:05:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:05:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:04:53 2024-09-16T13:05:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:05:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:05:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:05:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:05:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:05:13' 2024-09-16T13:05:18.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:05:18.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:05:18.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:05:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70270976.0,"last_snapshot_bytes":70270976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"} 2024-09-16T13:05:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:05:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:05:18.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:05:18.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:05:18.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:05:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:05:18.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:05:18.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:05:18.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:04:53' 2024-09-16T13:05:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-16 13:04:53' 2024-09-16T13:05:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:05:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:18.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:04:53' 2024-09-16T13:05:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:05:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:05:18.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:05:18.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK542544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70164480.0,"last_snapshot_bytes":70164480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:13image-primary6b18ebe6c-4efe-4743-8aee-94780a9c5a06up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70270976.0,"last_snapshot_bytes":70270976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:13image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70205440.0,"last_snapshot_bytes":70205440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:13image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70258688.0,"last_snapshot_bytes":70258688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:13image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70127616.0,"last_snapshot_bytes":70127616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:13' 2024-09-16T13:05:18.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:05:18.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T13:05:18.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:04:53' 2024-09-16T13:05:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T13:05:18.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:05:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:05:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:05:18.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:05:18.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:18.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:18.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-16T13:05:18.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:05:18.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:18.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:05:18.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-16T13:05:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-16T13:05:19.977 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:19.964+0000 7ff4f6946700 -1 reset not still connected to 0x557562001110 2024-09-16T13:05:20.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:05:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-16T13:05:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:05:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:20.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary6 2024-09-16T13:05:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:05:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:20.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary6 2024-09-16T13:05:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:05:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary7 2024-09-16T13:05:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-16T13:05:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:05:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:05:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:05:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:20.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:05:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-16T13:05:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:05:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:05:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:05:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:05:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:05:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T13:05:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:05:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: f491577d-20f7-4b88-9fb4-e72a02bb468c 2024-09-16T13:05:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:05:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70205440.0,"last_snapshot_bytes":70205440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"} 2024-09-16T13:05:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:05:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:04:53 2024-09-16T13:05:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:05:21.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:05:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:05:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:05:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:05:18' 2024-09-16T13:05:21.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:05:21.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:05:21.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:05:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70205440.0,"last_snapshot_bytes":70205440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"} 2024-09-16T13:05:21.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:05:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:05:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:05:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:05:21.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:05:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:05:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:05:21.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:05:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:04:53' 2024-09-16T13:05:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-16 13:04:53' 2024-09-16T13:05:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:05:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:04:53' 2024-09-16T13:05:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:05:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:05:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:05:21.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK442544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70164480.0,"last_snapshot_bytes":70164480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:18image-primary7f491577d-20f7-4b88-9fb4-e72a02bb468cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70205440.0,"last_snapshot_bytes":70205440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:18image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70258688.0,"last_snapshot_bytes":70258688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:18image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70127616.0,"last_snapshot_bytes":70127616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:04:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:18' 2024-09-16T13:05:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:05:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T13:05:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:04:53' 2024-09-16T13:05:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T13:05:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:05:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:05:21.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:05:21.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:05:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-16T13:05:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:05:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:05:21.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-16T13:05:21.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-16T13:05:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:05:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-16T13:05:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:05:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:24.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:05:24.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary7 2024-09-16T13:05:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:24.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary7 2024-09-16T13:05:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:05:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary8 2024-09-16T13:05:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-16T13:05:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:05:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:05:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:05:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:24.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:05:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-16T13:05:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:05:25.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:05:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:05:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:05:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:05:25.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T13:05:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:05:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: c7162764-8ecd-41dd-bf00-31141188bb23 2024-09-16T13:05:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:05:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70258688.0,"last_snapshot_bytes":70258688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"} 2024-09-16T13:05:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:05:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:05:23 2024-09-16T13:05:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:05:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:05:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:05:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:05:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:05:23' 2024-09-16T13:05:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:05:25.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:05:25.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:05:25.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70258688.0,"last_snapshot_bytes":70258688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"} 2024-09-16T13:05:25.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:05:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:05:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:05:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:05:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:05:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:05:25.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:05:25.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:05:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:05:23' 2024-09-16T13:05:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-16 13:05:23' 2024-09-16T13:05:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:05:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:05:23' 2024-09-16T13:05:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:05:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:05:25.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:05:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK342544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70164480.0,"last_snapshot_bytes":70164480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:05:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:23image-primary8c7162764-8ecd-41dd-bf00-31141188bb23up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70258688.0,"last_snapshot_bytes":70258688,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491801,"remote_snapshot_timestamp":1726491801,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:05:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:23image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70127616.0,"last_snapshot_bytes":70127616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:05:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:23' 2024-09-16T13:05:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:05:25.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T13:05:25.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:05:23' 2024-09-16T13:05:25.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T13:05:25.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:05:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:05:25.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:05:25.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:05:25.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-16T13:05:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:05:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:05:25.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-16T13:05:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-16T13:05:27.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:05:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-16T13:05:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:05:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:05:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary8 2024-09-16T13:05:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:27.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary8 2024-09-16T13:05:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:05:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary9 2024-09-16T13:05:27.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-16T13:05:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:05:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:05:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:05:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:27.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:05:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-16T13:05:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:05:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:05:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:05:28.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:05:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:05:28.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T13:05:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:05:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 18c6fc24-69f8-4190-a25c-6b81712dda2c 2024-09-16T13:05:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:05:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70127616.0,"last_snapshot_bytes":70127616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"} 2024-09-16T13:05:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:05:28.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:05:23 2024-09-16T13:05:28.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:05:28.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:05:28.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:05:28.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:05:28.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:05:23' 2024-09-16T13:05:28.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:05:28.147 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:05:28.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:05:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70127616.0,"last_snapshot_bytes":70127616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"} 2024-09-16T13:05:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:05:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:05:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:05:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:05:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:05:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:05:28.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:05:28.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:05:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:05:23' 2024-09-16T13:05:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-16 13:05:23' 2024-09-16T13:05:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:05:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:05:23' 2024-09-16T13:05:28.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:05:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:05:28.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:05:28.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK242544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70164480.0,"last_snapshot_bytes":70164480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:05:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:23image-primary918c6fc24-69f8-4190-a25c-6b81712dda2cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70127616.0,"last_snapshot_bytes":70127616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:05:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:23' 2024-09-16T13:05:28.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:05:28.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T13:05:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:05:23' 2024-09-16T13:05:28.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T13:05:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:05:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:05:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:05:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:05:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-16T13:05:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:05:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:28.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:05:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-16T13:05:28.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-16T13:05:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:05:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-16T13:05:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:05:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:05:29.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary9 2024-09-16T13:05:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary9 2024-09-16T13:05:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:05:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary10 2024-09-16T13:05:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-16T13:05:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:05:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:05:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:05:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:29.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:05:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-16T13:05:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:05:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:05:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:05:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:05:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:05:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T13:05:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:05:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7 2024-09-16T13:05:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:05:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70164480.0,"last_snapshot_bytes":70164480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"} 2024-09-16T13:05:31.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:05:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:05:23 2024-09-16T13:05:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:05:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:05:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:05:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:05:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:05:28' 2024-09-16T13:05:31.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:05:31.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:05:31.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:05:31.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70164480.0,"last_snapshot_bytes":70164480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"} 2024-09-16T13:05:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:05:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:05:31.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:05:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:05:31.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:05:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:05:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:05:31.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:05:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:05:23' 2024-09-16T13:05:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-16 13:05:23' 2024-09-16T13:05:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:05:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:05:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:05:23' 2024-09-16T13:05:31.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:05:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:05:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:05:31.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK142544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10b2fc05f6-b06d-47ba-bb56-bfa5cff1f0e7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70164480.0,"last_snapshot_bytes":70164480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726491802,"remote_snapshot_timestamp":1726491802,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:05:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:05:28' 2024-09-16T13:05:31.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:05:31.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T13:05:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:05:23' 2024-09-16T13:05:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T13:05:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:05:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:05:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:05:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:05:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-16T13:05:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:05:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:05:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-16T13:05:31.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-16T13:05:33.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:05:33.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-16T13:05:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:05:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:05:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:05:33.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary10 2024-09-16T13:05:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:05:33.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary10 2024-09-16T13:05:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..10} 2024-09-16T13:05:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS=() 2024-09-16T13:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS=() 2024-09-16T13:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS=() 2024-09-16T13:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS=() 2024-09-16T13:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ RET=0 2024-09-16T13:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary1 2024-09-16T13:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary1 2024-09-16T13:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-16T13:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:05:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:05:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:05:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-16T13:05:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:05:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:05:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:05:33.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:05:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:33.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-16T13:05:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:05:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-16T13:05:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:05:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:05:33.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-16T13:05:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:05:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-16T13:05:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary1': 2024-09-16T13:05:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:05:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:05:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:05:34.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 27fcd49101fc 2024-09-16T13:05:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.27fcd49101fc 2024-09-16T13:05:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:05:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:05:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:05:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:05:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:05:33 2024 2024-09-16T13:05:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:05:33 2024 2024-09-16T13:05:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:05:33 2024 2024-09-16T13:05:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:05:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:05:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: b5436ea9-dd4d-41f8-a883-e993197f4846 2024-09-16T13:05:34.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:05:34.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:05:34.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary1 2024-09-16T13:05:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd0 2024-09-16T13:05:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:05:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd0 2024-09-16T13:05:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:05:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:05:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:05:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: dd9ac1f2-6b03-4bb0-8177-a5232cb92f82 2024-09-16T13:05:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:05:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:05:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:05:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:05:34.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:05:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:05:35.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:05:35.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:05:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary1 2024-09-16T13:05:35.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd0 test-primary1 2024-09-16T13:05:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:05:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary1 2024-09-16T13:05:35.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:05:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-16T13:05:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:05:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:05:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:05:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:05:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary1 2024-09-16T13:05:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary2 2024-09-16T13:05:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:35.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary2 2024-09-16T13:05:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:05:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-16T13:05:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:05:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:05:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:05:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary1 2024-09-16T13:05:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:05:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:35.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-16T13:05:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-16T13:05:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:05:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:05:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:05:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:05:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:35.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-16T13:05:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:05:35.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-16T13:05:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:05:35.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-16T13:05:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:05:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:05:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-16T13:05:36.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3297 2024-09-16T13:05:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:36.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:05:36.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-16T13:05:36.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary2': 2024-09-16T13:05:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:05:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:05:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:05:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 280c1ec7fcce 2024-09-16T13:05:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.280c1ec7fcce 2024-09-16T13:05:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:05:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:05:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:05:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:05:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:05:35 2024 2024-09-16T13:05:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:05:35 2024 2024-09-16T13:05:36.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:05:35 2024 2024-09-16T13:05:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:05:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:05:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: aef3c5cc-43f2-49e8-a964-1dc4b1af8a46 2024-09-16T13:05:36.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:05:36.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:05:36.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary2 2024-09-16T13:05:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd1 2024-09-16T13:05:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:05:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd1 2024-09-16T13:05:36.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:05:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:05:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:05:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: e9085c74-89bf-4b44-b644-03c17731b7b8 2024-09-16T13:05:36.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:05:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:05:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:05:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:05:36.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:05:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:05:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:05:37.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:05:37.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary2 2024-09-16T13:05:37.157 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd1 test-primary2 2024-09-16T13:05:37.183 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:05:37.172+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T13:05:37.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:05:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary2 2024-09-16T13:05:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:05:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:05:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:05:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary3 2024-09-16T13:05:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary2 2024-09-16T13:05:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary3 2024-09-16T13:05:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-16T13:05:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary2 2024-09-16T13:05:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:05:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-16T13:05:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-16T13:05:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:05:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:05:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-16T13:05:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:05:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:05:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:05:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:05:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-16T13:05:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:05:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:05:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:05:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:05:37.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:05:37.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-16T13:05:37.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:37.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:37.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:05:37.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:05:37.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-16T13:05:37.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:05:37.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-16T13:05:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:05:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3300 2024-09-16T13:05:38.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-16T13:05:38.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary3': 2024-09-16T13:05:38.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:05:38.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:05:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:05:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 281ca66cae41 2024-09-16T13:05:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.281ca66cae41 2024-09-16T13:05:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:05:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:05:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:05:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:05:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:05:37 2024 2024-09-16T13:05:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:05:37 2024 2024-09-16T13:05:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:05:37 2024 2024-09-16T13:05:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:05:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:05:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 5d43d580-c61f-4fea-ad9b-9e4c3ad78c1f 2024-09-16T13:05:38.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:05:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:05:38.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary3 2024-09-16T13:05:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd2 2024-09-16T13:05:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:05:38.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd2 2024-09-16T13:05:38.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:05:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:05:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:05:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: abb41b64-5d59-4f15-9b74-5e95f7bc53a8 2024-09-16T13:05:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:05:38.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:05:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:05:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:05:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:05:39.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:05:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:05:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:05:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary3 2024-09-16T13:05:39.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd2 test-primary3 2024-09-16T13:05:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:05:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary3 2024-09-16T13:05:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:05:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-16T13:05:39.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:05:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:05:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:05:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:05:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:05:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:05:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary3 2024-09-16T13:05:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary4 2024-09-16T13:05:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary4 2024-09-16T13:05:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-16T13:05:39.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary3 2024-09-16T13:05:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-16T13:05:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:05:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:05:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:05:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-16T13:05:39.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:05:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:05:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:05:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:05:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:39.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-16T13:05:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:05:39.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:05:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:05:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:05:39.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-16T13:05:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:05:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:05:39.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-16T13:05:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:05:39.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-16T13:05:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3304 2024-09-16T13:05:40.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:40.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:05:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-16T13:05:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary4': 2024-09-16T13:05:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:05:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:05:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:05:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 282b3c9d70f6 2024-09-16T13:05:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.282b3c9d70f6 2024-09-16T13:05:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:05:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:05:40.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:05:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:05:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:05:39 2024 2024-09-16T13:05:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:05:39 2024 2024-09-16T13:05:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:05:39 2024 2024-09-16T13:05:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:05:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:05:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 46c5f7ce-56e5-4725-9f6f-d1c94f8c9c3e 2024-09-16T13:05:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:05:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:05:40.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary4 2024-09-16T13:05:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd3 2024-09-16T13:05:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:05:40.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd3 2024-09-16T13:05:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:05:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:05:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:05:40.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 31398750-3bb3-4b5d-9e2e-cab4466938ba 2024-09-16T13:05:40.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:05:40.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:05:40.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:05:40.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:05:40.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:05:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:05:41.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:05:41.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:05:41.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary4 2024-09-16T13:05:41.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd3 test-primary4 2024-09-16T13:05:41.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3305 2024-09-16T13:05:41.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:05:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:05:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:05:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:05:41.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary4 2024-09-16T13:05:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:05:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-16T13:05:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:05:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:05:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:05:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary5 2024-09-16T13:05:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary4 2024-09-16T13:05:41.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary5 2024-09-16T13:05:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-16T13:05:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:05:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:41.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:05:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary4 2024-09-16T13:05:41.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:05:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-16T13:05:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:05:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:05:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:05:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:41.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-16T13:05:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:05:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:41.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:05:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:05:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:05:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-16T13:05:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:05:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-16T13:05:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:05:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:05:41.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-16T13:05:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:05:41.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-16T13:05:42.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:05:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-16T13:05:42.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3309 2024-09-16T13:05:42.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary5': 2024-09-16T13:05:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:05:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:05:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:05:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 28413f7b572c 2024-09-16T13:05:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.28413f7b572c 2024-09-16T13:05:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:05:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:05:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:05:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:05:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:05:41 2024 2024-09-16T13:05:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:05:41 2024 2024-09-16T13:05:42.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:05:41 2024 2024-09-16T13:05:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:05:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:05:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 40484d62-c2e4-4d70-a7a6-1d77d8ae20f2 2024-09-16T13:05:42.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:05:42.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:05:42.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary5 2024-09-16T13:05:42.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd4 2024-09-16T13:05:42.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:05:42.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd4 2024-09-16T13:05:42.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:05:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:05:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:05:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 425ce0fa-f008-4dad-923c-7046ed979658 2024-09-16T13:05:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:05:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:05:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:05:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:05:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:05:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:05:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 Snapshot ID: 3308 2024-09-16T13:05:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T13:05:43.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:05:43.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary5 2024-09-16T13:05:43.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd4 test-primary5 2024-09-16T13:05:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:05:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:05:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:05:43.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:05:43.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary5 2024-09-16T13:05:43.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:05:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:05:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:05:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary5 2024-09-16T13:05:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary6 2024-09-16T13:05:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary6 2024-09-16T13:05:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-16T13:05:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary5 2024-09-16T13:05:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-16T13:05:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:05:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:43.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:05:43.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:05:43.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-16T13:05:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:05:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:05:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:05:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:05:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-16T13:05:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-16T13:05:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:05:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:05:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:05:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:05:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:05:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-16T13:05:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:05:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:05:43.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-16T13:05:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:05:43.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-16T13:05:44.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:05:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:05:44.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:05:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:05:44.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3313 2024-09-16T13:05:44.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:44.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-16T13:05:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary6': 2024-09-16T13:05:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:05:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:05:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:05:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 28532fef7a72 2024-09-16T13:05:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.28532fef7a72 2024-09-16T13:05:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:05:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:05:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:05:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:05:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:05:43 2024 2024-09-16T13:05:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:05:43 2024 2024-09-16T13:05:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:05:43 2024 2024-09-16T13:05:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:05:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:05:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 3483ebd3-b9b7-4f5f-86db-00f691453f56 2024-09-16T13:05:44.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:05:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:05:44.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary6 2024-09-16T13:05:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd5 2024-09-16T13:05:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:05:44.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd5 2024-09-16T13:05:44.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:05:44.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:05:44.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:05:44.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: fbc1871a-e974-4bd8-bff8-7580ad53ae58 2024-09-16T13:05:44.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:05:44.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:05:44.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:05:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:05:44.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:05:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:05:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:05:44.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:05:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary6 2024-09-16T13:05:44.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd5 test-primary6 2024-09-16T13:05:45.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:05:45.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary6 2024-09-16T13:05:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:05:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-16T13:05:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:05:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:05:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:45.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:05:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:05:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:05:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:05:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary7 2024-09-16T13:05:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary7 2024-09-16T13:05:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-16T13:05:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary6 2024-09-16T13:05:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:05:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary6 2024-09-16T13:05:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:05:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-16T13:05:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:05:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-16T13:05:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:05:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:05:45.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:05:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:05:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:45.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-16T13:05:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:05:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-16T13:05:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:05:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:05:45.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-16T13:05:45.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:05:45.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-16T13:05:45.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3312 2024-09-16T13:05:45.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:45.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3319 2024-09-16T13:05:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:45.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:45.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:05:45.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:05:45.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:05:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:05:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:05:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:05:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3318 2024-09-16T13:05:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:05:46.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-16T13:05:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary7': 2024-09-16T13:05:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:05:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:05:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:05:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 2863872b4790 2024-09-16T13:05:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.2863872b4790 2024-09-16T13:05:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:05:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:05:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:05:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:05:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:05:45 2024 2024-09-16T13:05:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:05:45 2024 2024-09-16T13:05:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:05:45 2024 2024-09-16T13:05:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:05:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:05:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 270aa00f-3fdb-4e45-9116-ea259ca6b320 2024-09-16T13:05:46.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:05:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:05:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary7 2024-09-16T13:05:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd6 2024-09-16T13:05:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:05:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd6 2024-09-16T13:05:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:05:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:05:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:05:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 162537e9-f3ad-45f2-b1d8-7f69a150b0f4 2024-09-16T13:05:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:05:46.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:05:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:05:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:05:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:05:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:05:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:05:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:05:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary7 2024-09-16T13:05:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd6 test-primary7 2024-09-16T13:05:47.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3323 2024-09-16T13:05:47.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:05:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:05:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:05:47.322 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:05:47.312+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T13:05:47.323 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:05:47.312+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T13:05:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:05:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary7 2024-09-16T13:05:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:05:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-16T13:05:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:05:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:05:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:05:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:05:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary8 2024-09-16T13:05:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary8 2024-09-16T13:05:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:05:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:05:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-16T13:05:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:47.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-16T13:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:05:47.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:05:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:05:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:05:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-16T13:05:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary7 2024-09-16T13:05:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary7 2024-09-16T13:05:47.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-16T13:05:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:05:47.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-16T13:05:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:05:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:05:47.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-16T13:05:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:05:47.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-16T13:05:48.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3324 2024-09-16T13:05:48.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:48.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3328 2024-09-16T13:05:48.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:05:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:05:48.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:05:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:05:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:05:48.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:05:48.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:05:48.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-16T13:05:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary8': 2024-09-16T13:05:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:05:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:05:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:05:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 287dd48d5a7e 2024-09-16T13:05:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.287dd48d5a7e 2024-09-16T13:05:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:05:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:05:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:05:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:05:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:05:47 2024 2024-09-16T13:05:48.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:05:47 2024 2024-09-16T13:05:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:05:47 2024 2024-09-16T13:05:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:05:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:05:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: d2b71a04-786f-44dd-bc9a-3d01975d02f7 2024-09-16T13:05:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:05:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:05:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary8 2024-09-16T13:05:48.543 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:48.532+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1748f0 2024-09-16T13:05:48.544 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:48.532+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174d00 2024-09-16T13:05:48.544 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:48.532+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T13:05:48.561 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:48.552+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1748f0 2024-09-16T13:05:48.562 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:48.552+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174d00 2024-09-16T13:05:48.562 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:48.552+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T13:05:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd7 2024-09-16T13:05:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:05:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd7 2024-09-16T13:05:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:05:48.755 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T13:05:48.740+0000 7f4071c2f700 -1 reset not still connected to 0x563591d7d040 2024-09-16T13:05:48.755 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T13:05:48.744+0000 7f4071c2f700 -1 reset not still connected to 0x56359d5a2680 2024-09-16T13:05:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:05:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:05:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 5a56af20-c3da-44e1-9cde-41711aeffc80 2024-09-16T13:05:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:05:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:05:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:05:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:05:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:05:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:05:49.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:05:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:05:49.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:05:49.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 Snapshot ID: 3327 2024-09-16T13:05:49.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T13:05:49.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:05:49.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary8 2024-09-16T13:05:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd7 test-primary8 2024-09-16T13:05:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:05:49.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary8 2024-09-16T13:05:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:05:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-16T13:05:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:05:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:05:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:05:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:05:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:05:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:05:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary8 2024-09-16T13:05:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary9 2024-09-16T13:05:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary9 2024-09-16T13:05:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-16T13:05:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary8 2024-09-16T13:05:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-16T13:05:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:05:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:05:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:05:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-16T13:05:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:05:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:05:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:05:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:05:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:49.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-16T13:05:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:05:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-16T13:05:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:05:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-16T13:05:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:05:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:05:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-16T13:05:50.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3331 2024-09-16T13:05:50.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3332 2024-09-16T13:05:50.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:50.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:05:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:05:50.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:05:50.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3336 2024-09-16T13:05:50.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:05:50.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:50.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-16T13:05:50.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary9': 2024-09-16T13:05:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:05:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:05:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:05:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 2894b9d5afbf 2024-09-16T13:05:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.2894b9d5afbf 2024-09-16T13:05:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:05:50.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:05:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:05:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:05:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:05:49 2024 2024-09-16T13:05:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:05:49 2024 2024-09-16T13:05:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:05:49 2024 2024-09-16T13:05:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:05:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:05:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 3041a084-27f4-42db-87ac-d71aabb63b76 2024-09-16T13:05:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:05:50.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:05:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary9 2024-09-16T13:05:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd8 2024-09-16T13:05:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:05:50.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd8 2024-09-16T13:05:50.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:05:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:05:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:05:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 0b2ccb83-aeba-4250-91c0-cc3928e37dea 2024-09-16T13:05:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:05:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:05:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:05:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:05:50.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:05:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:05:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:05:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:05:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary9 2024-09-16T13:05:51.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd8 test-primary9 2024-09-16T13:05:51.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:05:51.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary9 2024-09-16T13:05:51.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:05:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:05:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:05:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary10 2024-09-16T13:05:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary10 2024-09-16T13:05:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-16T13:05:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:05:51.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:05:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:05:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-16T13:05:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:05:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:05:51.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:05:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:05:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:05:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-16T13:05:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary9 2024-09-16T13:05:51.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary9 2024-09-16T13:05:51.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-16T13:05:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-16T13:05:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:05:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:05:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:05:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:05:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:05:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:51.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:05:51.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:05:51.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3333 2024-09-16T13:05:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:05:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:05:51.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:05:51.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:05:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-16T13:05:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:05:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:05:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-16T13:05:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:05:51.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-16T13:05:52.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3340 2024-09-16T13:05:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:05:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:52.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:05:52.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:05:52.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:05:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3342 2024-09-16T13:05:52.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:52.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-16T13:05:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary10': 2024-09-16T13:05:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:05:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:05:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:05:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 28a61253c8b5 2024-09-16T13:05:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.28a61253c8b5 2024-09-16T13:05:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:05:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:05:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:05:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:05:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:05:51 2024 2024-09-16T13:05:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:05:51 2024 2024-09-16T13:05:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:05:51 2024 2024-09-16T13:05:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:05:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:05:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: b6a5803f-d8e8-4bd6-8abc-6f43c1584d8a 2024-09-16T13:05:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:05:52.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:05:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary10 2024-09-16T13:05:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd9 2024-09-16T13:05:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:05:52.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd9 2024-09-16T13:05:52.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:05:52.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:05:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:05:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: f4f4e958-0ede-4fb1-a5df-ba1347959c37 2024-09-16T13:05:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:05:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:05:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:05:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:05:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:05:53.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:05:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:05:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:05:53.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary10 2024-09-16T13:05:53.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd9 test-primary10 2024-09-16T13:05:53.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3344 2024-09-16T13:05:53.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3345 2024-09-16T13:05:53.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:53.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:05:53.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary10 2024-09-16T13:05:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:05:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:53.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:05:53.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:05:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:05:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:05:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:05:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:05:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:05:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:05:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 86824 2024-09-16T13:05:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary10 2024-09-16T13:05:53.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary10 2024-09-16T13:05:53.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-16T13:05:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-16T13:05:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:05:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:05:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:05:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:05:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:05:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:05:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:05:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:05:53.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-16T13:05:54.039 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:54.028+0000 7ff4f6946700 -1 reset not still connected to 0x5575585dc750 2024-09-16T13:05:54.039 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:54.028+0000 7ff4f6946700 -1 reset not still connected to 0x557558f289c0 2024-09-16T13:05:54.039 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:54.028+0000 7ff4f6946700 -1 reset not still connected to 0x55755c9e2340 2024-09-16T13:05:54.039 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:54.032+0000 7ff4f6946700 -1 reset not still connected to 0x55756936cf70 2024-09-16T13:05:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:05:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:05:54.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:05:54.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3350 2024-09-16T13:05:54.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3353 2024-09-16T13:05:54.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:55.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3355 2024-09-16T13:05:55.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:55.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:05:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:05:55.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:05:55.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3352 2024-09-16T13:05:55.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:55.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3354 2024-09-16T13:05:55.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:05:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:05:55.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:05:55.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:05:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:05:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:05:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:05:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:05:56.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:05:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3359 2024-09-16T13:05:56.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:57.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3361 2024-09-16T13:05:57.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3362 2024-09-16T13:05:57.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:57.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:05:57.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:57.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:57.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:05:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:05:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:05:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:05:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:05:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:58.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:05:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:05:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:05:58.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3364 2024-09-16T13:05:58.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3363 2024-09-16T13:05:58.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:05:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:05:58.360 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:05:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:05:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:05:58.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:05:59.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3367 2024-09-16T13:05:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3370 2024-09-16T13:05:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:05:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:05:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:05:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:05:59.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:05:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:05:59.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:05:59.344 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:59.336+0000 7ff4f6946700 -1 reset not still connected to 0x557556316dd0 2024-09-16T13:05:59.357 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:59.340+0000 7ff4f6946700 -1 reset not still connected to 0x5575585dc750 2024-09-16T13:05:59.359 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:59.340+0000 7ff4f6946700 -1 reset not still connected to 0x557558f289c0 2024-09-16T13:05:59.359 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:59.340+0000 7ff4f6946700 -1 reset not still connected to 0x55755c9e2340 2024-09-16T13:05:59.359 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:05:59.344+0000 7ff4f6946700 -1 reset not still connected to 0x55756936cf70 2024-09-16T13:06:00.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3373 2024-09-16T13:06:00.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3375 2024-09-16T13:06:00.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3374 2024-09-16T13:06:00.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:00.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:06:00.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:00.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:06:00.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:06:00.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:06:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:06:00.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:06:00.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3376 2024-09-16T13:06:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:06:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:06:01.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:06:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:06:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:06:01.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:06:02.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3382 2024-09-16T13:06:02.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:02.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:02.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:06:02.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:02.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:02.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:06:02.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:06:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:06:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:06:02.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:06:02.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3381 2024-09-16T13:06:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:03.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3384 2024-09-16T13:06:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3383 2024-09-16T13:06:03.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:03.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:06:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:06:03.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:06:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:06:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:06:03.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:06:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:06:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:06:03.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:06:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3390 2024-09-16T13:06:04.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3388 2024-09-16T13:06:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:06:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:06:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:06:05.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3393 2024-09-16T13:06:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3394 2024-09-16T13:06:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:05.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:05.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3395 2024-09-16T13:06:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:05.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:05.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:06:05.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:05.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:05.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:06:05.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:06:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:06:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:06:05.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:06:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3399 2024-09-16T13:06:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:06:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:06:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:06:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:06:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:06:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:06:07.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3402 2024-09-16T13:06:07.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:06:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:06:07.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:06:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3401 2024-09-16T13:06:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:06:07.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:06:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:06:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:07.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:06:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:06:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:06:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:06:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:06:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:06:08.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3406 2024-09-16T13:06:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:08.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3405 2024-09-16T13:06:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:06:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:06:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:06:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:06:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:06:09.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:06:09.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3409 2024-09-16T13:06:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:09.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3407 2024-09-16T13:06:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:10.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3412 2024-09-16T13:06:10.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3413 2024-09-16T13:06:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:06:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:06:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:06:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:10.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3415 2024-09-16T13:06:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:10.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:06:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:06:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:06:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:06:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:06:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:06:11.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3419 2024-09-16T13:06:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:06:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:06:11.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:06:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:12.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3421 2024-09-16T13:06:12.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3422 2024-09-16T13:06:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:12.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:06:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:06:12.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:06:12.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:12.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:06:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:06:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:06:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3425 2024-09-16T13:06:13.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:13.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3426 2024-09-16T13:06:13.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:06:13.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:06:13.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:06:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:06:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:06:13.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:06:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:06:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:06:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:06:13.878 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T13:06:13.867+0000 7f4071c2f700 -1 reset not still connected to 0x5635b9608680 2024-09-16T13:06:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:06:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:06:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:06:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3429 2024-09-16T13:06:14.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3430 2024-09-16T13:06:14.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:14.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3433 2024-09-16T13:06:15.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3435 2024-09-16T13:06:15.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3434 2024-09-16T13:06:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:06:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:06:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:06:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:06:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:06:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:06:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:15.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:15.840 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:06:15.827+0000 7ff4f6946700 -1 reset not still connected to 0x55756936cf70 2024-09-16T13:06:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:06:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:06:16.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:06:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:06:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:06:16.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:06:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3436 2024-09-16T13:06:16.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:17.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:06:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:06:17.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:06:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:17.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:06:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:06:17.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:06:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3441 2024-09-16T13:06:17.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3442 2024-09-16T13:06:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:06:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:06:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:06:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:18.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:06:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:06:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:06:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:06:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:06:18.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:06:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3446 2024-09-16T13:06:18.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3443 2024-09-16T13:06:18.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:18.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:06:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:06:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:06:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3447 2024-09-16T13:06:19.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3449 2024-09-16T13:06:19.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:19.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:20.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3455 2024-09-16T13:06:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:20.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3452 2024-09-16T13:06:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:06:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:06:20.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:06:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:06:20.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:06:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:06:20.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3454 2024-09-16T13:06:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:20.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:21.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3457 2024-09-16T13:06:21.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:06:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:06:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:06:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:06:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:21.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:06:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:06:22.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3462 2024-09-16T13:06:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:06:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:22.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:06:22.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:06:22.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3461 2024-09-16T13:06:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:06:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:06:22.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:06:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3466 2024-09-16T13:06:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:23.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3465 2024-09-16T13:06:23.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:23.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:06:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:06:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:06:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:06:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:06:23.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:06:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:06:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:06:23.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:06:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:06:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:06:24.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:06:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3470 2024-09-16T13:06:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3469 2024-09-16T13:06:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3473 2024-09-16T13:06:25.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3474 2024-09-16T13:06:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:06:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:06:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:06:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3475 2024-09-16T13:06:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:06:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:06:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:06:25.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:25.989 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:06:25.980+0000 7f74def64700 -1 reset not still connected to 0x559441c00a90 2024-09-16T13:06:25.990 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:06:25.980+0000 7f74def64700 -1 reset not still connected to 0x55945e6385b0 2024-09-16T13:06:25.990 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:06:25.980+0000 7f74def64700 -1 reset not still connected to 0x55946478dad0 2024-09-16T13:06:25.990 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:06:25.980+0000 7f74def64700 -1 reset not still connected to 0x559465716340 2024-09-16T13:06:25.990 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:06:25.980+0000 7f74def64700 -1 reset not still connected to 0x5594657172b0 2024-09-16T13:06:25.990 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:06:25.980+0000 7f74def64700 -1 reset not still connected to 0x559466ed8680 2024-09-16T13:06:25.990 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:06:25.980+0000 7f74def64700 -1 reset not still connected to 0x559467c91930 2024-09-16T13:06:25.990 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:06:25.980+0000 7f74def64700 -1 reset not still connected to 0x55946d1d5ba0 2024-09-16T13:06:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:06:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:06:26.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:06:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:06:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:06:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:06:26.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3479 2024-09-16T13:06:26.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:27.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3482 2024-09-16T13:06:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:06:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:06:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:06:27.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3481 2024-09-16T13:06:27.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:27.360 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:06:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:06:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:06:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:28.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:06:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:06:28.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:06:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:06:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:06:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:06:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3486 2024-09-16T13:06:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:06:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:28.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:06:28.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:06:28.360 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:28.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3483 2024-09-16T13:06:28.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:29.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3489 2024-09-16T13:06:29.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3490 2024-09-16T13:06:29.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:06:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:06:29.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:06:29.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:06:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:06:30.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:06:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:06:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:06:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:06:30.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3494 2024-09-16T13:06:30.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3495 2024-09-16T13:06:30.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3493 2024-09-16T13:06:30.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:30.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:30.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3499 2024-09-16T13:06:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:06:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:06:31.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:06:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:06:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:06:31.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:06:31.922 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:06:31.912+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e36aa000 2024-09-16T13:06:31.922 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:06:31.912+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e36ab1e0 2024-09-16T13:06:31.936 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:06:31.924+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e5d9d2b0 2024-09-16T13:06:31.937 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:06:31.924+0000 7f315ffe6700 -1 reset not still connected to 0x55c204aa35f0 2024-09-16T13:06:31.937 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:06:31.924+0000 7f315ffe6700 -1 reset not still connected to 0x55c207203930 2024-09-16T13:06:32.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3500 2024-09-16T13:06:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3501 2024-09-16T13:06:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:06:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:32.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:06:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:06:32.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:06:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:32.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:06:32.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:06:33.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3505 2024-09-16T13:06:33.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3504 2024-09-16T13:06:33.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:33.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:06:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:06:33.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:06:33.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:06:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:06:33.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:06:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:06:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:06:33.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:06:34.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3509 2024-09-16T13:06:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3510 2024-09-16T13:06:34.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:06:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:34.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:06:34.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:06:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3515 2024-09-16T13:06:35.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3514 2024-09-16T13:06:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:35.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3513 2024-09-16T13:06:35.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:06:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:06:35.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:06:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:06:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:35.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:06:35.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:06:36.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3519 2024-09-16T13:06:36.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:36.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:06:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:06:36.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:06:36.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:36.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:06:36.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:36.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:36.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:06:36.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:06:37.233 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:06:37.223+0000 7ff4f6946700 -1 reset not still connected to 0x557556316dd0 2024-09-16T13:06:37.233 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:06:37.223+0000 7ff4f6946700 -1 reset not still connected to 0x557557d0b860 2024-09-16T13:06:37.233 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:06:37.223+0000 7ff4f6946700 -1 reset not still connected to 0x5575585dc750 2024-09-16T13:06:37.233 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:06:37.223+0000 7ff4f6946700 -1 reset not still connected to 0x557558f289c0 2024-09-16T13:06:37.237 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:06:37.223+0000 7ff4f6946700 -1 reset not still connected to 0x55755c9e2340 2024-09-16T13:06:37.243 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:06:37.231+0000 7ff4f6946700 -1 reset not still connected to 0x55756936cf70 2024-09-16T13:06:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:06:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:06:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:06:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3522 2024-09-16T13:06:37.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:06:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:06:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:06:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3521 2024-09-16T13:06:37.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3525 2024-09-16T13:06:38.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3526 2024-09-16T13:06:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:06:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:06:38.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:06:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:06:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:06:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:06:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:38.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:38.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:38.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:06:38.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:38.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:38.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:06:38.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:06:39.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:06:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:06:39.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:06:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3530 2024-09-16T13:06:39.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3529 2024-09-16T13:06:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3533 2024-09-16T13:06:40.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3532 2024-09-16T13:06:40.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3534 2024-09-16T13:06:40.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:06:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:06:40.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:06:40.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:40.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:40.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:06:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:06:40.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:06:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3536 2024-09-16T13:06:41.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:06:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:06:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:06:41.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:41.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:06:41.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:41.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:41.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:06:41.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:06:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3541 2024-09-16T13:06:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3542 2024-09-16T13:06:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:06:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:06:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:06:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:06:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:06:42.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:06:42.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3545 2024-09-16T13:06:43.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3544 2024-09-16T13:06:43.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:06:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:06:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:06:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:06:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:06:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:06:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:06:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:06:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:06:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:06:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:06:44.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:06:44.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3549 2024-09-16T13:06:44.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3550 2024-09-16T13:06:44.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:44.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:06:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:06:45.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:06:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:06:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3554 2024-09-16T13:06:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:06:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:06:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3555 2024-09-16T13:06:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:45.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3553 2024-09-16T13:06:45.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:45.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:46.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:46.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:06:46.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:46.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:46.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:06:46.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:06:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:06:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:06:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:06:46.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3556 2024-09-16T13:06:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:47.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3561 2024-09-16T13:06:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:06:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:06:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:06:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:06:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:47.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:06:47.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:06:47.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3562 2024-09-16T13:06:47.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3564 2024-09-16T13:06:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:06:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:06:48.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:06:48.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:06:48.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:06:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:06:48.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3563 2024-09-16T13:06:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:06:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:06:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:06:48.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:06:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:49.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:06:49.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:06:49.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3570 2024-09-16T13:06:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3569 2024-09-16T13:06:49.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:49.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:06:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:06:50.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:06:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3573 2024-09-16T13:06:50.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3574 2024-09-16T13:06:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:06:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:06:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:06:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:50.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3575 2024-09-16T13:06:50.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:50.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:06:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:51.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:06:51.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:06:51.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3576 2024-09-16T13:06:51.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:06:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:06:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:06:52.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3580 2024-09-16T13:06:52.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:06:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:06:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:06:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:06:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:06:52.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:06:52.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:52.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3582 2024-09-16T13:06:52.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:53.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3585 2024-09-16T13:06:53.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3586 2024-09-16T13:06:53.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:06:53.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:06:53.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:06:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:53.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:06:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:06:53.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:06:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:06:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:06:53.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:06:54.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3589 2024-09-16T13:06:54.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:54.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:06:54.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:06:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:06:54.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3590 2024-09-16T13:06:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:55.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3595 2024-09-16T13:06:55.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3593 2024-09-16T13:06:55.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:06:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:06:55.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:06:55.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:06:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:06:55.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:06:55.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:06:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:06:56.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:06:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:06:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:06:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:06:56.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3596 2024-09-16T13:06:56.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3599 2024-09-16T13:06:56.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:56.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3601 2024-09-16T13:06:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:06:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:06:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:06:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:06:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:06:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:06:57.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3603 2024-09-16T13:06:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:57.929 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:06:57.915+0000 7ff4f6946700 -1 reset not still connected to 0x5575585dc750 2024-09-16T13:06:57.929 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:06:57.915+0000 7ff4f6946700 -1 reset not still connected to 0x557558f289c0 2024-09-16T13:06:57.929 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:06:57.915+0000 7ff4f6946700 -1 reset not still connected to 0x55755c9e2340 2024-09-16T13:06:57.932 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:06:57.923+0000 7ff4f6946700 -1 reset not still connected to 0x55756936cf70 2024-09-16T13:06:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:06:58.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:06:58.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:06:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:06:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:06:58.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:06:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3604 2024-09-16T13:06:58.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3605 2024-09-16T13:06:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:58.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:59.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3610 2024-09-16T13:06:59.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3611 2024-09-16T13:06:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:06:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:06:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:06:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:06:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:06:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:06:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:06:59.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:06:59.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:06:59.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:06:59.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:00.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3612 2024-09-16T13:07:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:07:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:07:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:07:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:07:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:07:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:07:00.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:00.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3613 2024-09-16T13:07:00.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:01.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3619 2024-09-16T13:07:01.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3618 2024-09-16T13:07:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:07:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:07:01.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:07:01.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:07:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:07:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:07:02.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3622 2024-09-16T13:07:02.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3621 2024-09-16T13:07:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:07:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:07:02.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:07:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:07:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:07:02.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:07:02.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:02.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:03.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3627 2024-09-16T13:07:03.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3625 2024-09-16T13:07:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:03.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:07:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:03.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:07:03.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:07:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:07:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:07:03.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:07:04.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3630 2024-09-16T13:07:04.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3628 2024-09-16T13:07:04.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:04.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:04.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:07:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:07:04.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:07:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:07:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:07:04.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:07:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3635 2024-09-16T13:07:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3634 2024-09-16T13:07:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:05.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:07:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:07:05.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:07:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:07:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:07:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:07:06.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3638 2024-09-16T13:07:06.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3639 2024-09-16T13:07:06.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:07:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:07:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:07:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:07:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:07:06.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:07:07.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3643 2024-09-16T13:07:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3642 2024-09-16T13:07:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:07:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:07:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:07:07.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:07.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:07:07.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:07.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:07.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:07:07.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:07:07.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:07.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3646 2024-09-16T13:07:08.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3647 2024-09-16T13:07:08.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:07:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:07:08.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:07:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:07:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:07:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:07:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:08.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3651 2024-09-16T13:07:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:07:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:07:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:07:09.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3650 2024-09-16T13:07:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:07:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:07:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:07:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:09.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3655 2024-09-16T13:07:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:07:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:07:10.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:07:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:07:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:07:10.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:07:10.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3654 2024-09-16T13:07:10.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3657 2024-09-16T13:07:11.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3658 2024-09-16T13:07:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:07:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:07:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:07:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:07:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:07:11.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:07:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:11.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:07:12.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:07:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:07:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:07:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:07:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:07:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3663 2024-09-16T13:07:12.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3662 2024-09-16T13:07:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3667 2024-09-16T13:07:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3666 2024-09-16T13:07:13.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:13.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:07:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:07:13.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:07:13.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:07:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:07:13.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:07:14.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3670 2024-09-16T13:07:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:07:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:07:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:07:14.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3669 2024-09-16T13:07:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:07:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:07:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:07:14.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:14.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:15.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3675 2024-09-16T13:07:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3674 2024-09-16T13:07:15.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:15.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:07:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:07:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:07:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:07:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:07:15.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:07:16.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:07:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:07:16.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:07:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:07:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:07:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:07:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3679 2024-09-16T13:07:16.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3678 2024-09-16T13:07:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:07:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:07:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:07:17.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3682 2024-09-16T13:07:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:17.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:07:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:07:17.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:07:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3683 2024-09-16T13:07:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:07:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:07:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:07:18.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:07:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:07:18.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:07:18.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3684 2024-09-16T13:07:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3685 2024-09-16T13:07:18.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:18.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:07:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:07:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:19.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:07:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:07:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:07:19.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:07:19.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3691 2024-09-16T13:07:19.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3689 2024-09-16T13:07:19.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:19.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:20.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3694 2024-09-16T13:07:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:07:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:07:20.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:07:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:07:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:07:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:07:20.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:20.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3695 2024-09-16T13:07:20.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:07:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:07:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:07:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:07:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:07:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:07:21.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3697 2024-09-16T13:07:21.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3696 2024-09-16T13:07:21.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:21.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3702 2024-09-16T13:07:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3703 2024-09-16T13:07:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:07:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:07:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:07:22.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:22.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:07:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:07:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:07:22.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:23.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3706 2024-09-16T13:07:23.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3707 2024-09-16T13:07:23.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:23.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:07:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:23.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:07:23.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:07:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:07:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:07:23.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:07:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3710 2024-09-16T13:07:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3711 2024-09-16T13:07:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:24.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:07:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:07:24.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:07:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:07:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:07:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:07:25.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3715 2024-09-16T13:07:25.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3714 2024-09-16T13:07:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:07:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:07:25.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:07:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:07:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:07:25.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:07:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:07:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:07:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:07:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:07:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:07:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:07:26.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3719 2024-09-16T13:07:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3718 2024-09-16T13:07:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:26.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:07:27.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:07:27.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:07:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:07:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:07:27.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:07:27.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3722 2024-09-16T13:07:27.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3723 2024-09-16T13:07:27.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:27.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:28.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:07:28.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:28.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:28.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:07:28.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:07:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:07:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:07:28.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:07:28.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3724 2024-09-16T13:07:28.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3725 2024-09-16T13:07:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:28.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:07:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:07:29.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:07:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:07:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:07:29.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:07:29.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3728 2024-09-16T13:07:29.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3729 2024-09-16T13:07:29.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:29.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:30.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3734 2024-09-16T13:07:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:07:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:07:30.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:07:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:07:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:07:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:07:30.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3735 2024-09-16T13:07:30.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:30.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:31.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3739 2024-09-16T13:07:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:07:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:07:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:07:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:07:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:07:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:07:31.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:31.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3738 2024-09-16T13:07:31.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:32.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:07:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:07:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:07:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:07:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:07:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:07:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3743 2024-09-16T13:07:32.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3742 2024-09-16T13:07:32.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:32.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3746 2024-09-16T13:07:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3745 2024-09-16T13:07:33.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:33.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:33.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:07:33.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:33.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:33.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:07:33.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:07:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:07:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:07:33.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:07:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3750 2024-09-16T13:07:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3751 2024-09-16T13:07:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:34.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:07:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:07:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:07:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:07:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:07:34.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:07:35.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3754 2024-09-16T13:07:35.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3755 2024-09-16T13:07:35.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:07:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:07:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:07:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:07:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:07:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:07:36.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3758 2024-09-16T13:07:36.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3759 2024-09-16T13:07:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:07:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:07:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:07:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:07:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:07:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:07:36.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:07:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:07:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:07:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:07:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:37.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:07:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:07:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3762 2024-09-16T13:07:37.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3763 2024-09-16T13:07:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:07:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:07:38.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:07:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:07:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:07:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:07:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3767 2024-09-16T13:07:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3766 2024-09-16T13:07:38.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:38.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:39.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:07:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:07:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:07:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:07:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:07:39.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:07:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3769 2024-09-16T13:07:39.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3768 2024-09-16T13:07:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:39.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:40.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3774 2024-09-16T13:07:40.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3775 2024-09-16T13:07:40.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:07:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:07:40.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:07:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:07:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:40.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:07:40.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:07:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3776 2024-09-16T13:07:41.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3778 2024-09-16T13:07:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:07:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:07:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:07:41.313 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:41.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:07:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:07:41.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:07:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:07:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:07:42.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:07:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3782 2024-09-16T13:07:42.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3783 2024-09-16T13:07:42.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:42.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:07:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:07:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:07:43.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3786 2024-09-16T13:07:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3787 2024-09-16T13:07:43.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:43.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:07:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:07:43.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:07:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:07:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:07:43.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:07:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:44.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:07:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:07:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:07:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3790 2024-09-16T13:07:44.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3791 2024-09-16T13:07:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:07:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:07:44.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:07:44.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:44.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:45.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3795 2024-09-16T13:07:45.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3794 2024-09-16T13:07:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:07:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:07:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:07:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:07:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:07:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:07:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:07:46.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:07:46.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:07:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:07:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:46.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:07:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:07:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3799 2024-09-16T13:07:46.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3798 2024-09-16T13:07:46.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:07:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:07:47.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:07:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:07:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:07:47.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:07:47.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3803 2024-09-16T13:07:47.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3802 2024-09-16T13:07:47.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:47.360 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3804 2024-09-16T13:07:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:07:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:07:48.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:07:48.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:07:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:07:48.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:07:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3805 2024-09-16T13:07:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:49.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:07:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:07:49.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:07:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:07:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:07:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:07:49.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3810 2024-09-16T13:07:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3811 2024-09-16T13:07:49.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:49.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:50.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3814 2024-09-16T13:07:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:07:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:50.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:07:50.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:07:50.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:07:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:07:50.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:07:50.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:50.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3815 2024-09-16T13:07:50.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:07:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:07:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:07:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:07:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:07:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:07:51.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3819 2024-09-16T13:07:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3818 2024-09-16T13:07:51.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:52.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3822 2024-09-16T13:07:52.374 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:07:52.366+0000 7ff4f6946700 -1 reset not still connected to 0x55756dda71e0 2024-09-16T13:07:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:07:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:07:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:07:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:07:52.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:07:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:07:52.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3823 2024-09-16T13:07:52.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:52.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:53.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3827 2024-09-16T13:07:53.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3825 2024-09-16T13:07:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:53.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:07:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:07:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:07:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:53.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:07:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:07:53.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:07:54.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3830 2024-09-16T13:07:54.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3831 2024-09-16T13:07:54.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:07:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:07:54.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:07:54.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:54.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:07:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:07:54.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:07:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:07:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:55.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:07:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:07:55.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3834 2024-09-16T13:07:55.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3835 2024-09-16T13:07:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:07:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:07:55.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:07:55.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:07:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:07:56.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:07:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:07:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:07:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:07:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3837 2024-09-16T13:07:56.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3838 2024-09-16T13:07:56.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:57.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3842 2024-09-16T13:07:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:07:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:07:57.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:07:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:07:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:57.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:07:57.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:07:57.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3843 2024-09-16T13:07:57.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:57.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:58.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3844 2024-09-16T13:07:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:07:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:07:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:07:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3845 2024-09-16T13:07:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:07:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:07:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:07:58.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:59.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:07:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:07:59.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:07:59.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3851 2024-09-16T13:07:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:07:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:07:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:07:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:07:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:07:59.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:07:59.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3850 2024-09-16T13:07:59.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:07:59.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3854 2024-09-16T13:08:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:08:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:08:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:08:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:08:00.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:00.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:00.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:08:00.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:08:00.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:00.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3855 2024-09-16T13:08:00.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:01.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:08:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:08:01.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:08:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:08:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:08:01.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:08:01.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3858 2024-09-16T13:08:01.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3859 2024-09-16T13:08:01.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:08:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:02.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:08:02.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:08:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:08:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:02.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:08:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:08:02.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3863 2024-09-16T13:08:02.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:03.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3867 2024-09-16T13:08:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3866 2024-09-16T13:08:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3864 2024-09-16T13:08:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:03.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:03.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:03.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:08:03.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:03.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:03.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:08:03.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:08:03.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:08:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:08:03.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:08:04.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3872 2024-09-16T13:08:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3871 2024-09-16T13:08:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:08:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:08:04.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:08:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:08:04.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 86958 2024-09-16T13:08:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3876 2024-09-16T13:08:05.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3875 2024-09-16T13:08:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:08:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:08:05.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:08:06.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3879 2024-09-16T13:08:06.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:08:06.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:08:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:08:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:08:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 87092 2024-09-16T13:08:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:08:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:08:06.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:08:07.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3881 2024-09-16T13:08:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:08:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:08:07.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:08:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:08:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:08:07.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:08:07.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:08.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3884 2024-09-16T13:08:08.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:08:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 87268 2024-09-16T13:08:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:08:08.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:08:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:08:08.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3883 2024-09-16T13:08:08.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:08.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3887 2024-09-16T13:08:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:09.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:08:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:08:09.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:08:09.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3888 2024-09-16T13:08:09.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:10.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3891 2024-09-16T13:08:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:08:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:08:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:08:11.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3892 2024-09-16T13:08:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:08:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 87423 2024-09-16T13:08:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:08:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:08:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:08:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:08:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:08:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:08:12.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3895 2024-09-16T13:08:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:08:12.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 87577 2024-09-16T13:08:13.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3897 2024-09-16T13:08:13.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:08:13.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 87777 2024-09-16T13:08:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3899 2024-09-16T13:08:14.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:08:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:08:14.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:08:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:08:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:08:15.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:08:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:08:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 87973 2024-09-16T13:08:16.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3901 2024-09-16T13:08:16.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:08:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:08:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:08:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:08:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:08:17.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:08:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3903 2024-09-16T13:08:17.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:19.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:08:19.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 88129 2024-09-16T13:08:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3905 2024-09-16T13:08:19.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:08:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:08:20.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 88329 2024-09-16T13:08:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T13:08:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:08:22.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 86825 2024-09-16T13:10:35.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:10:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:10:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:10:35.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 86959 2024-09-16T13:10:37.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:10:37.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:10:37.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:10:37.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 87093 2024-09-16T13:10:39.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:10:39.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:10:39.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:10:39.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 87269 2024-09-16T13:10:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:10:41.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:10:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:10:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 87424 2024-09-16T13:10:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:10:43.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:10:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:10:43.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 87578 2024-09-16T13:10:45.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:10:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:10:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:10:45.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 87778 2024-09-16T13:10:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:10:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:10:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:10:47.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 87974 2024-09-16T13:10:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:10:49.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:10:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:10:49.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 88130 2024-09-16T13:10:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:10:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:10:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:10:51.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 88330 2024-09-16T13:10:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:10:53.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:10:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T13:10:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:10:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:10:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:10:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 1 2024-09-16T13:10:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:10:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd0 2024-09-16T13:10:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary1 2024-09-16T13:10:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary1 2024-09-16T13:10:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:10:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:10:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary1 2024-09-16T13:10:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 2 2024-09-16T13:10:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:10:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd1 2024-09-16T13:10:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary2 2024-09-16T13:10:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary2 2024-09-16T13:10:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:10:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:10:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary2 2024-09-16T13:10:53.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 3 2024-09-16T13:10:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd2 2024-09-16T13:10:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary3 2024-09-16T13:10:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:10:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary3 2024-09-16T13:10:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:10:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary3 2024-09-16T13:10:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:10:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 4 2024-09-16T13:10:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:10:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:10:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd3 2024-09-16T13:10:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary4 2024-09-16T13:10:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary4 2024-09-16T13:10:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:10:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary4 2024-09-16T13:10:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:10:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:10:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 6 2024-09-16T13:10:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:10:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:10:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 7 2024-09-16T13:10:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd5 2024-09-16T13:10:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary6 2024-09-16T13:10:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary6 2024-09-16T13:10:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:10:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary6 2024-09-16T13:10:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd6 2024-09-16T13:10:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:10:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:10:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary7 2024-09-16T13:10:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary7 2024-09-16T13:10:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 8 2024-09-16T13:10:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:10:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary7 2024-09-16T13:10:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:10:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:10:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd7 2024-09-16T13:10:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary8 2024-09-16T13:10:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary8 2024-09-16T13:10:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:10:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary8 2024-09-16T13:10:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 5 2024-09-16T13:10:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd4 2024-09-16T13:10:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 9 2024-09-16T13:10:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary5 2024-09-16T13:10:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:10:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary5 2024-09-16T13:10:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:10:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary5 2024-09-16T13:10:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd8 2024-09-16T13:10:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:10:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 94181 2024-09-16T13:10:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary9 2024-09-16T13:10:53.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary9 2024-09-16T13:10:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:10:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary9 2024-09-16T13:10:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 10 2024-09-16T13:10:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd9 2024-09-16T13:10:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary10 2024-09-16T13:10:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary10 2024-09-16T13:10:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:10:53.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary10 2024-09-16T13:10:53.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:10:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd1 2024-09-16T13:10:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:10:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd5 2024-09-16T13:10:53.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:10:53.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd6 2024-09-16T13:10:53.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd3 2024-09-16T13:10:53.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:10:54.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:10:54.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd9 2024-09-16T13:10:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:10:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd2 2024-09-16T13:10:54.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:10:54.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd8 2024-09-16T13:10:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:10:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd4 2024-09-16T13:10:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:10:54.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd7 2024-09-16T13:10:54.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:10:54.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd0 2024-09-16T13:11:48.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=17f422ebf3553ca03fe3ec995693490e 2024-09-16T13:11:48.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary7 2024-09-16T13:11:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=a593739ed96b4e49d7f61b487225ab1d 2024-09-16T13:11:49.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary9 2024-09-16T13:11:50.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=bbfe5bf77f3b8245349a677237a6e228 2024-09-16T13:11:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary5 2024-09-16T13:11:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=82a6c97fac9850063740a9c671c24f31 2024-09-16T13:11:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary4 2024-09-16T13:11:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=9bf6b851de2e731ef3f797026d085420 2024-09-16T13:11:50.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary6 2024-09-16T13:11:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=f43cc1294aa9d4bf63c44e1edf6b3fb8 2024-09-16T13:11:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary3 2024-09-16T13:11:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=b9dba56418371379778301c4e217d8a1 2024-09-16T13:11:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary2 2024-09-16T13:11:51.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-16T13:11:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:11:51.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-16T13:11:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-16T13:11:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:11:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-16T13:11:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=160351907f4d028f8885cb900840c98f 2024-09-16T13:11:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary8 2024-09-16T13:11:51.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-16T13:11:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:11:51.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-16T13:11:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-16T13:11:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:51.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:11:51.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-16T13:11:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-16T13:11:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:11:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-16T13:11:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-16T13:11:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:52.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:11:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-16T13:11:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=777386ebf05dfa7bc3f54689dda595ba 2024-09-16T13:11:52.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary1 2024-09-16T13:11:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-16T13:11:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:11:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-16T13:11:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=a109c98dad14c97ba3aa94448a076c43 2024-09-16T13:11:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary10 2024-09-16T13:11:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-16T13:11:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:52.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:11:52.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-16T13:11:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-16T13:11:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:11:52.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-16T13:11:52.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-16T13:11:52.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:52.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:52.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:11:52.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-16T13:11:53.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:11:53.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:11:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-16T13:11:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:11:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:53.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:11:53.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:11:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-16T13:11:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:53.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:11:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:11:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:11:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:11:53.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:11:53.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:11:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-16T13:11:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:53.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:11:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:11:53.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:11:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-16T13:11:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:11:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:11:53.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:11:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-16T13:11:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:11:53.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:11:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:11:54.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:11:54.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:11:54.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:11:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-16T13:11:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:11:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:54.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:11:54.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:11:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-16T13:11:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:11:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:11:54.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:11:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:11:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:11:54.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-16T13:11:54.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:11:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:11:54.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:11:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-16T13:11:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:11:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:11:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:11:54.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-16T13:11:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:11:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:11:54.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:11:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-16T13:11:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-16T13:11:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:11:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:11:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:54.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-16T13:11:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-16T13:11:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:11:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:11:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-16T13:11:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:11:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:54.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T13:11:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:54.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T13:11:54.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:54.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T13:11:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T13:11:54.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:54.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T13:11:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:11:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 3483ebd3-b9b7-4f5f-86db-00f691453f56 2024-09-16T13:11:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:11:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:11:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:53 2024-09-16T13:11:54.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:11:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:11:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:11:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:54' 2024-09-16T13:11:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:11:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:11:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 270aa00f-3fdb-4e45-9116-ea259ca6b320 2024-09-16T13:11:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:11:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:11:54.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:53 2024-09-16T13:11:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:11:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:11:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:11:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:54' 2024-09-16T13:11:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:11:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:11:54.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 3041a084-27f4-42db-87ac-d71aabb63b76 2024-09-16T13:11:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:11:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:11:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:53 2024-09-16T13:11:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:11:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:11:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:11:54.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:54' 2024-09-16T13:11:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:11:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:11:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 5d43d580-c61f-4fea-ad9b-9e4c3ad78c1f 2024-09-16T13:11:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:11:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:11:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:53 2024-09-16T13:11:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:11:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:11:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:11:54.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:54' 2024-09-16T13:11:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:11:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:11:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 40484d62-c2e4-4d70-a7a6-1d77d8ae20f2 2024-09-16T13:11:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:11:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:11:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:53 2024-09-16T13:11:54.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:11:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:11:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:11:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:54' 2024-09-16T13:11:54.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:11:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:11:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:11:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:11:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:11:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:11:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:11:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:11:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:11:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:11:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:11:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:11:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:11:55.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:11:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:11:55.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:11:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:11:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:11:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:11:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:11:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:11:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:11:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:11:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:11:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:55.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:11:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:11:55.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:11:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:11:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:11:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:11:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:11:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:11:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:11:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:11:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:11:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:11:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:11:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:11:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:11:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:11:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:11:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:11:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:11:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:11:55.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:11:55.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:11:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-16T13:11:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:11:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T13:11:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-16T13:11:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:11:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:55.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:55.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T13:11:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-16T13:11:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:11:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:55.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:55.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:55.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T13:11:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-16T13:11:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:11:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T13:11:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-16T13:11:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:11:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:55.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:55.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T13:11:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:11:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: aef3c5cc-43f2-49e8-a964-1dc4b1af8a46 2024-09-16T13:11:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:11:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:11:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:53 2024-09-16T13:11:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:11:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:11:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:11:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:55' 2024-09-16T13:11:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:11:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:11:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:11:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:11:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b5436ea9-dd4d-41f8-a883-e993197f4846 2024-09-16T13:11:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:11:55.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:11:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:53 2024-09-16T13:11:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:11:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:11:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:11:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:55' 2024-09-16T13:11:55.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:11:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:11:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:11:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:53' 2024-09-16T13:11:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:53' 2024-09-16T13:11:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-16 13:11:53' 2024-09-16T13:11:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-16 13:11:53' 2024-09-16T13:11:55.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:11:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:11:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:11:53' 2024-09-16T13:11:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:11:53' 2024-09-16T13:11:55.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:11:55.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:11:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:11:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:53' 2024-09-16T13:11:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-16 13:11:53' 2024-09-16T13:11:55.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:53' 2024-09-16T13:11:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-16 13:11:53' 2024-09-16T13:11:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:11:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:11:53' 2024-09-16T13:11:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:11:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:11:53' 2024-09-16T13:11:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:55.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:53' 2024-09-16T13:11:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-16 13:11:53' 2024-09-16T13:11:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:11:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:11:53' 2024-09-16T13:11:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:55.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:11:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:11:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:11:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:11:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:11:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:11:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:11:55.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:11:55.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:55.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:11:55.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:11:55.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:11:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:11:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 46c5f7ce-56e5-4725-9f6f-d1c94f8c9c3e 2024-09-16T13:11:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:11:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:11:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:53 2024-09-16T13:11:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:11:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:11:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:11:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:55' 2024-09-16T13:11:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:11:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:11:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:11:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: d2b71a04-786f-44dd-bc9a-3d01975d02f7 2024-09-16T13:11:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:11:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:11:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:53 2024-09-16T13:11:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:11:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:11:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:11:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:55' 2024-09-16T13:11:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:11:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:11:55.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:11:55.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:11:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:11:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:11:55.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:11:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:11:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:11:55.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:11:55.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:11:55.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:11:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:53' 2024-09-16T13:11:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-16 13:11:53' 2024-09-16T13:11:55.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:11:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:11:53' 2024-09-16T13:11:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:11:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:53' 2024-09-16T13:11:55.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:11:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-16 13:11:53' 2024-09-16T13:11:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:11:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:11:53' 2024-09-16T13:11:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:55.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:11:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:11:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:11:55.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:11:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:11:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:11:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:11:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:55.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:11:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:53' 2024-09-16T13:11:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-16 13:11:53' 2024-09-16T13:11:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:11:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:11:53' 2024-09-16T13:11:55.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:55.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:11:55.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:11:55.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:11:55.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:11:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:53' 2024-09-16T13:11:55.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-16 13:11:53' 2024-09-16T13:11:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:11:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:11:53' 2024-09-16T13:11:55.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:55.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:11:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:11:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b6a5803f-d8e8-4bd6-8abc-6f43c1584d8a 2024-09-16T13:11:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:11:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:11:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:53 2024-09-16T13:11:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:11:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:11:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:11:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:55' 2024-09-16T13:11:55.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:11:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:11:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:11:55.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:11:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:11:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:11:55.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:55.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:11:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:11:55.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:11:55.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:11:55.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:53' 2024-09-16T13:11:55.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-16 13:11:53' 2024-09-16T13:11:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:11:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:11:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:11:53' 2024-09-16T13:11:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:11:55.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54' 2024-09-16T13:11:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:11:55.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T13:11:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54' 2024-09-16T13:11:55.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:11:55.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T13:11:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54' 2024-09-16T13:11:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:11:55.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T13:11:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54' 2024-09-16T13:11:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:11:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T13:11:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54' 2024-09-16T13:11:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:11:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T13:11:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54' 2024-09-16T13:11:55.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:11:55.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T13:11:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54' 2024-09-16T13:11:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:11:55.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T13:11:55.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54' 2024-09-16T13:11:55.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:11:55.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T13:11:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54' 2024-09-16T13:11:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:11:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T13:11:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:55image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:11:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:11:54' 2024-09-16T13:11:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:11:55.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T13:11:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:53' 2024-09-16T13:11:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:53' 2024-09-16T13:11:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:53' 2024-09-16T13:11:56.181 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:53' 2024-09-16T13:11:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:53' 2024-09-16T13:11:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T13:11:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:53' 2024-09-16T13:11:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:53' 2024-09-16T13:11:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T13:11:56.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T13:11:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:53' 2024-09-16T13:11:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T13:11:56.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T13:11:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T13:11:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:53' 2024-09-16T13:11:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T13:11:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T13:11:56.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T13:11:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:53' 2024-09-16T13:11:56.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T13:11:56.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:11:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:56.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:11:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:11:56.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:11:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:11:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:11:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:11:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:11:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:11:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:56.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:11:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:11:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:11:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:11:56.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:11:56.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:11:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:11:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-16T13:11:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:11:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:11:56.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:11:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:11:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:56.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:11:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:56.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-16T13:11:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:11:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:56.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:11:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:11:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-16T13:11:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:11:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:56.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:11:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:11:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-16T13:11:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:11:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:11:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:11:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-16T13:11:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:11:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:11:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:11:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-16T13:11:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:11:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:56.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:11:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:11:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-16T13:11:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:11:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-16T13:11:56.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:11:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:11:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:11:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:11:56.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:11:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:11:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-16T13:11:56.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:11:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:11:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:11:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:11:56.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-16T13:11:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:11:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:11:56.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:11:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-16T13:11:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:11:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:57.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-16T13:11:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:11:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-16T13:11:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-16T13:11:57.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:11:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:11:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-16T13:11:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:11:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-16T13:11:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-16T13:11:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-16T13:11:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-16T13:11:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:11:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:11:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:57.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-16T13:11:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:11:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-16T13:11:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-16T13:11:57.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-16T13:11:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:11:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-16T13:11:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-16T13:11:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-16T13:11:57.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:11:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:11:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:11:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-16T13:11:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-16T13:11:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-16T13:11:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-16T13:11:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:11:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 46c5f7ce-56e5-4725-9f6f-d1c94f8c9c3e 2024-09-16T13:11:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:11:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:11:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:55 2024-09-16T13:11:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:11:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:11:57.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:11:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:53' 2024-09-16T13:11:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:11:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:11:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 270aa00f-3fdb-4e45-9116-ea259ca6b320 2024-09-16T13:11:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:11:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:11:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:54 2024-09-16T13:11:57.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:11:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:11:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:11:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:53' 2024-09-16T13:11:57.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:11:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:11:57.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:11:57.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:11:57.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:11:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:11:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 40484d62-c2e4-4d70-a7a6-1d77d8ae20f2 2024-09-16T13:11:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:11:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:11:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:54 2024-09-16T13:11:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:11:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:11:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:11:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:53' 2024-09-16T13:11:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:11:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:11:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:11:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:11:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:11:57.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:11:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:11:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:11:57.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:11:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:11:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:11:57.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:11:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:11:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:11:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:11:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: aef3c5cc-43f2-49e8-a964-1dc4b1af8a46 2024-09-16T13:11:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:57.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:11:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:11:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:55 2024-09-16T13:11:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:11:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:11:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:11:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:53' 2024-09-16T13:11:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:11:57.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:57.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:11:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:11:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:11:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:11:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b5436ea9-dd4d-41f8-a883-e993197f4846 2024-09-16T13:11:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:57.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:11:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:11:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:55 2024-09-16T13:11:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:11:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:11:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:11:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:53' 2024-09-16T13:11:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:11:57.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:54' 2024-09-16T13:11:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-16 13:11:54' 2024-09-16T13:11:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:11:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:11:54' 2024-09-16T13:11:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:11:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:11:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: d2b71a04-786f-44dd-bc9a-3d01975d02f7 2024-09-16T13:11:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:11:57.316 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:11:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:55 2024-09-16T13:11:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:11:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:11:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:11:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:53' 2024-09-16T13:11:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:11:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:57.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:11:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:11:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:11:57.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:11:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:11:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:11:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:11:57.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:11:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:11:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:11:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:11:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:11:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:11:57.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:11:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:11:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:11:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:11:57.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:11:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:55' 2024-09-16T13:11:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-16 13:11:55' 2024-09-16T13:11:57.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:11:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:11:55' 2024-09-16T13:11:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:57.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:11:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:11:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:11:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:11:57.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:11:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:11:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 5d43d580-c61f-4fea-ad9b-9e4c3ad78c1f 2024-09-16T13:11:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:11:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:11:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:54 2024-09-16T13:11:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:11:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:11:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:11:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:53' 2024-09-16T13:11:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:11:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:11:57.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:11:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:11:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 3483ebd3-b9b7-4f5f-86db-00f691453f56 2024-09-16T13:11:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:11:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:11:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:54 2024-09-16T13:11:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:11:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:11:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:11:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:53' 2024-09-16T13:11:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:11:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:11:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:11:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:11:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:11:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:55' 2024-09-16T13:11:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-16 13:11:55' 2024-09-16T13:11:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:11:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:11:55' 2024-09-16T13:11:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:55' 2024-09-16T13:11:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:11:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-16 13:11:55' 2024-09-16T13:11:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:11:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:11:55' 2024-09-16T13:11:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:11:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:11:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b6a5803f-d8e8-4bd6-8abc-6f43c1584d8a 2024-09-16T13:11:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:11:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:11:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:55 2024-09-16T13:11:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:11:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:11:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:11:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:53' 2024-09-16T13:11:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:11:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:11:57.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:11:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 3041a084-27f4-42db-87ac-d71aabb63b76 2024-09-16T13:11:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:11:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:11:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:54 2024-09-16T13:11:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:11:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:11:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:11:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:11:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:11:53' 2024-09-16T13:11:57.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:11:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:11:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:11:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:11:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:11:57.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:11:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:11:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:11:57.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:54' 2024-09-16T13:11:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-16 13:11:54' 2024-09-16T13:11:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:11:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:11:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:11:54' 2024-09-16T13:11:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:11:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:11:57.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:11:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:11:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:11:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:11:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:57.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:11:57.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:11:57.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:11:57.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:11:57.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:11:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:11:57.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:11:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:11:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:11:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:11:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:55' 2024-09-16T13:11:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-16 13:11:55' 2024-09-16T13:11:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:11:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:11:55' 2024-09-16T13:11:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:11:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:54' 2024-09-16T13:11:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-16 13:11:54' 2024-09-16T13:11:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:11:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:11:54' 2024-09-16T13:11:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:11:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:11:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:11:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:11:57.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:11:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:11:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:11:57.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:11:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:11:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:11:57.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:11:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:57.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:57.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:11:57.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:57.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:11:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:54' 2024-09-16T13:11:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-16 13:11:54' 2024-09-16T13:11:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:11:57.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:11:54' 2024-09-16T13:11:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:11:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:11:57.360 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:11:57.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:11:57.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:11:57.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:57.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:11:57.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:11:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:54' 2024-09-16T13:11:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-16 13:11:54' 2024-09-16T13:11:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:11:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:11:54' 2024-09-16T13:11:57.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:55' 2024-09-16T13:11:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-16 13:11:55' 2024-09-16T13:11:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:11:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:11:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:11:55' 2024-09-16T13:11:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:11:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:11:57.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:11:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:11:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53' 2024-09-16T13:11:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:11:57.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T13:11:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:55' 2024-09-16T13:11:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T13:11:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:11:57.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:11:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-16T13:11:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:11:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:11:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-16T13:11:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53' 2024-09-16T13:11:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:11:57.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T13:11:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:55' 2024-09-16T13:11:57.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T13:11:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:11:57.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:57.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:11:57.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-16T13:11:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:11:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:11:57.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-16T13:11:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53' 2024-09-16T13:11:57.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:11:57.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T13:11:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:55' 2024-09-16T13:11:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T13:11:57.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:11:57.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:11:57.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:57.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:57.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-16T13:11:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:11:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:11:57.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-16T13:11:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53' 2024-09-16T13:11:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:11:57.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T13:11:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53' 2024-09-16T13:11:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:11:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T13:11:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53' 2024-09-16T13:11:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:11:57.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:55' 2024-09-16T13:11:57.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T13:11:57.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T13:11:57.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:54' 2024-09-16T13:11:57.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T13:11:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:54' 2024-09-16T13:11:57.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T13:11:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:11:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:11:57.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-16T13:11:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:11:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:11:57.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-16T13:11:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:11:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:11:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:57.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:11:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:57.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:11:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53' 2024-09-16T13:11:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:11:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:57.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-16T13:11:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:11:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:11:57.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-16T13:11:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53' 2024-09-16T13:11:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:11:57.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T13:11:57.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:57.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:57.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:57.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-16T13:11:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:11:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:11:57.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-16T13:11:57.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:54' 2024-09-16T13:11:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T13:11:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53' 2024-09-16T13:11:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:11:57.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T13:11:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T13:11:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:54' 2024-09-16T13:11:57.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T13:11:57.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:11:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:11:57.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:55cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:11:54cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:11:53' 2024-09-16T13:11:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:11:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:11:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:57.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T13:11:57.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:55' 2024-09-16T13:11:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T13:11:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:11:57.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:11:54' 2024-09-16T13:11:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T13:11:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:11:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:11:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:57.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-16T13:11:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:11:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:11:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-16T13:11:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:11:57.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:11:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-16T13:11:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:11:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:11:57.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-16T13:11:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:57.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-16T13:11:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:11:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:11:57.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-16T13:11:57.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:11:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:11:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:11:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-16T13:11:57.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:11:57.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:11:57.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:11:57.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:11:57.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-16T13:11:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:11:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:11:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:11:58.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:11:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:11:58.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:11:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:11:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:11:58.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:11:58.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary2 2024-09-16T13:11:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:11:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:11:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary9 2024-09-16T13:11:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary5 2024-09-16T13:11:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:11:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:11:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:11:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:11:58.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary4 2024-09-16T13:11:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary8 2024-09-16T13:11:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:11:58.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary6 2024-09-16T13:11:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:11:58.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary3 2024-09-16T13:11:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:11:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:11:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary10 2024-09-16T13:11:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:11:58.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary7 2024-09-16T13:11:58.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary1 2024-09-16T13:11:58.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd0 2024-09-16T13:11:58.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:11:58.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd0 2024-09-16T13:11:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd1 2024-09-16T13:11:58.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:11:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd1 2024-09-16T13:11:58.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd2 2024-09-16T13:11:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd2 2024-09-16T13:11:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:11:58.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd3 2024-09-16T13:11:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:11:58.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd3 2024-09-16T13:11:58.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd4 2024-09-16T13:11:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd4 2024-09-16T13:11:58.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:11:58.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd5 2024-09-16T13:11:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:11:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd5 2024-09-16T13:11:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd6 2024-09-16T13:11:58.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:11:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd6 2024-09-16T13:11:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd7 2024-09-16T13:11:58.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:11:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd7 2024-09-16T13:11:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd8 2024-09-16T13:11:58.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:11:58.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd8 2024-09-16T13:11:58.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd9 2024-09-16T13:11:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd9 2024-09-16T13:11:58.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:11:58.961 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:11:58.955+0000 7f315ffe6700 -1 reset not still connected to 0x55c1f51b44e0 2024-09-16T13:11:58.961 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:11:58.955+0000 7f315ffe6700 -1 reset not still connected to 0x55c1fe0afc70 2024-09-16T13:13:02.051 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:13:02.042+0000 7f74def64700 -1 reset not still connected to 0x55945c6c4a90 2024-09-16T13:13:35.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=a109c98dad14c97ba3aa94448a076c43 2024-09-16T13:13:35.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd8 2024-09-16T13:13:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=a593739ed96b4e49d7f61b487225ab1d 2024-09-16T13:13:35.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-09-16T13:13:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=bbfe5bf77f3b8245349a677237a6e228 2024-09-16T13:13:35.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd1 2024-09-16T13:13:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=b9dba56418371379778301c4e217d8a1 2024-09-16T13:13:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd2 2024-09-16T13:13:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=160351907f4d028f8885cb900840c98f 2024-09-16T13:13:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd6 2024-09-16T13:13:36.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=17f422ebf3553ca03fe3ec995693490e 2024-09-16T13:13:36.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd4 2024-09-16T13:13:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ a109c98dad14c97ba3aa94448a076c43 != \a\1\0\9\c\9\8\d\a\d\1\4\c\9\7\b\a\3\a\a\9\4\4\4\8\a\0\7\6\c\4\3 ]] 2024-09-16T13:13:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=9bf6b851de2e731ef3f797026d085420 2024-09-16T13:13:36.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd9 2024-09-16T13:13:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ a593739ed96b4e49d7f61b487225ab1d != \a\5\9\3\7\3\9\e\d\9\6\b\4\e\4\9\d\7\f\6\1\b\4\8\7\2\2\5\a\b\1\d ]] 2024-09-16T13:13:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=82a6c97fac9850063740a9c671c24f31 2024-09-16T13:13:36.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd5 2024-09-16T13:13:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ bbfe5bf77f3b8245349a677237a6e228 != \b\b\f\e\5\b\f\7\7\f\3\b\8\2\4\5\3\4\9\a\6\7\7\2\3\7\a\6\e\2\2\8 ]] 2024-09-16T13:13:37.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ b9dba56418371379778301c4e217d8a1 != \b\9\d\b\a\5\6\4\1\8\3\7\1\3\7\9\7\7\8\3\0\1\c\4\e\2\1\7\d\8\a\1 ]] 2024-09-16T13:13:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=f43cc1294aa9d4bf63c44e1edf6b3fb8 2024-09-16T13:13:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd7 2024-09-16T13:13:37.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 160351907f4d028f8885cb900840c98f != \1\6\0\3\5\1\9\0\7\f\4\d\0\2\8\f\8\8\8\5\c\b\9\0\0\8\4\0\c\9\8\f ]] 2024-09-16T13:13:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 17f422ebf3553ca03fe3ec995693490e != \1\7\f\4\2\2\e\b\f\3\5\5\3\c\a\0\3\f\e\3\e\c\9\9\5\6\9\3\4\9\0\e ]] 2024-09-16T13:13:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=777386ebf05dfa7bc3f54689dda595ba 2024-09-16T13:13:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd3 2024-09-16T13:13:37.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 9bf6b851de2e731ef3f797026d085420 != \9\b\f\6\b\8\5\1\d\e\2\e\7\3\1\e\f\3\f\7\9\7\0\2\6\d\0\8\5\4\2\0 ]] 2024-09-16T13:13:37.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 82a6c97fac9850063740a9c671c24f31 != \8\2\a\6\c\9\7\f\a\c\9\8\5\0\0\6\3\7\4\0\a\9\c\6\7\1\c\2\4\f\3\1 ]] 2024-09-16T13:13:37.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ f43cc1294aa9d4bf63c44e1edf6b3fb8 != \f\4\3\c\c\1\2\9\4\a\a\9\d\4\b\f\6\3\c\4\4\e\1\e\d\f\6\b\3\f\b\8 ]] 2024-09-16T13:13:37.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 777386ebf05dfa7bc3f54689dda595ba != \7\7\7\3\8\6\e\b\f\0\5\d\f\a\7\b\c\3\f\5\4\6\8\9\d\d\a\5\9\5\b\a ]] 2024-09-16T13:13:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:13:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 94182 2024-09-16T13:13:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:13:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 94183 2024-09-16T13:13:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:13:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 94186 2024-09-16T13:13:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:13:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 94188 2024-09-16T13:13:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:13:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 94189 2024-09-16T13:13:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:13:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 94191 2024-09-16T13:13:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:13:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 94192 2024-09-16T13:13:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:13:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 94194 2024-09-16T13:13:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:13:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 94197 2024-09-16T13:13:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T13:13:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:13:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary1 2024-09-16T13:13:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-16T13:13:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:37.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:13:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:13:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:13:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:13:37.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:13:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-16T13:13:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:13:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:13:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:13:39.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:13:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:13:39.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T13:13:39.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:13:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b5436ea9-dd4d-41f8-a883-e993197f4846 2024-09-16T13:13:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:13:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70856704.0,"last_snapshot_bytes":70856704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"} 2024-09-16T13:13:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:13:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:13:23 2024-09-16T13:13:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:13:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:13:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:13:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:13:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:13:38' 2024-09-16T13:13:39.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:13:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:13:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:13:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70856704.0,"last_snapshot_bytes":70856704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"} 2024-09-16T13:13:39.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:13:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:13:39.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:13:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:13:39.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:13:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:13:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:13:39.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:13:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:13:23' 2024-09-16T13:13:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-16 13:13:23' 2024-09-16T13:13:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:13:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:13:23' 2024-09-16T13:13:39.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:13:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:13:39.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:13:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1b5436ea9-dd4d-41f8-a883-e993197f4846up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70856704.0,"last_snapshot_bytes":70856704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:38image-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70103040.0,"last_snapshot_bytes":70103040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:38image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63688704.0,"last_snapshot_bytes":63688704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:38image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":114139136.0,"last_snapshot_bytes":114139136,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:38image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63909888.0,"last_snapshot_bytes":63909888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:38image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":114204672.0,"last_snapshot_bytes":114204672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:38image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65126400.0,"last_snapshot_bytes":65126400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:38image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62160896.0,"last_snapshot_bytes":62160896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:38image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:38image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:38' 2024-09-16T13:13:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:13:39.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T13:13:39.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:13:23' 2024-09-16T13:13:39.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T13:13:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:13:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:13:39.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:13:39.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:13:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-16T13:13:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:13:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:13:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-16T13:13:39.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-16T13:13:41.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:13:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-16T13:13:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:13:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:13:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:13:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary1 2024-09-16T13:13:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:41.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary1 2024-09-16T13:13:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:13:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary2 2024-09-16T13:13:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-16T13:13:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:13:41.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:13:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:13:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:13:41.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:13:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-16T13:13:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:13:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:13:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:13:42.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:13:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:13:42.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T13:13:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:13:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: aef3c5cc-43f2-49e8-a964-1dc4b1af8a46 2024-09-16T13:13:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:13:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63688704.0,"last_snapshot_bytes":63688704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"} 2024-09-16T13:13:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:13:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:13:23 2024-09-16T13:13:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:13:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:13:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:13:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:13:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:13:38' 2024-09-16T13:13:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:13:42.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:13:42.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:13:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63688704.0,"last_snapshot_bytes":63688704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"} 2024-09-16T13:13:42.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:13:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:13:42.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:13:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:13:42.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:13:42.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:13:42.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:13:42.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:13:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:13:23' 2024-09-16T13:13:42.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-16 13:13:23' 2024-09-16T13:13:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:13:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:13:23' 2024-09-16T13:13:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:13:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:13:42.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:13:42.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK942544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70103040.0,"last_snapshot_bytes":70103040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:38image-primary2aef3c5cc-43f2-49e8-a964-1dc4b1af8a46up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63688704.0,"last_snapshot_bytes":63688704,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:38image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":114139136.0,"last_snapshot_bytes":114139136,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:38image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63909888.0,"last_snapshot_bytes":63909888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:38image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":114204672.0,"last_snapshot_bytes":114204672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:38image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65126400.0,"last_snapshot_bytes":65126400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:38image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62160896.0,"last_snapshot_bytes":62160896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:38image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:38image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:38' 2024-09-16T13:13:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:13:42.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T13:13:42.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:13:23' 2024-09-16T13:13:42.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T13:13:42.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:13:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:13:42.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:13:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:13:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-16T13:13:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:13:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:13:42.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-16T13:13:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-16T13:13:43.807 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T13:13:43.795+0000 7f4071c2f700 -1 reset not still connected to 0x563591d7d040 2024-09-16T13:13:44.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:13:44.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-16T13:13:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:13:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:13:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:13:44.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary2 2024-09-16T13:13:44.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:44.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary2 2024-09-16T13:13:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:13:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary3 2024-09-16T13:13:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-16T13:13:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:13:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:13:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:13:44.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:13:44.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:13:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-16T13:13:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:13:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:13:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:13:45.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:13:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:13:45.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T13:13:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:13:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 5d43d580-c61f-4fea-ad9b-9e4c3ad78c1f 2024-09-16T13:13:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:13:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":114139136.0,"last_snapshot_bytes":114139136,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"} 2024-09-16T13:13:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:13:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:13:23 2024-09-16T13:13:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:13:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:13:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:13:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:13:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:13:43' 2024-09-16T13:13:45.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:13:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:13:45.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:13:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":114139136.0,"last_snapshot_bytes":114139136,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"} 2024-09-16T13:13:45.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:13:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:13:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:13:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:13:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:13:45.303 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:13:45.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:13:45.305 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:13:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:13:23' 2024-09-16T13:13:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-16 13:13:23' 2024-09-16T13:13:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:13:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:13:23' 2024-09-16T13:13:45.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:13:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:13:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:13:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK842544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70103040.0,"last_snapshot_bytes":70103040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:43image-primary35d43d580-c61f-4fea-ad9b-9e4c3ad78c1fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":114139136.0,"last_snapshot_bytes":114139136,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:43image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63909888.0,"last_snapshot_bytes":63909888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:43image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":114204672.0,"last_snapshot_bytes":114204672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:43image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65126400.0,"last_snapshot_bytes":65126400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:43image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62160896.0,"last_snapshot_bytes":62160896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:43image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:43image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:43' 2024-09-16T13:13:45.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:13:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T13:13:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:13:23' 2024-09-16T13:13:45.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T13:13:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:13:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:13:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:13:45.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:13:45.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:45.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:45.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-16T13:13:45.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:13:45.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:45.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:13:45.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-16T13:13:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-16T13:13:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:13:47.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-16T13:13:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:13:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:13:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:13:47.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary3 2024-09-16T13:13:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:47.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary3 2024-09-16T13:13:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:13:47.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary4 2024-09-16T13:13:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-16T13:13:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:13:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:13:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:13:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:13:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:13:47.400 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:13:47.387+0000 7f61b1855700 -1 reset not still connected to 0x557b1f47fd40 2024-09-16T13:13:47.408 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:13:47.391+0000 7f61b1855700 -1 reset not still connected to 0x557b229664e0 2024-09-16T13:13:47.408 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:13:47.391+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T13:13:47.408 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:13:47.395+0000 7f61b1855700 -1 reset not still connected to 0x557b3b101c70 2024-09-16T13:13:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-16T13:13:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:13:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:13:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:13:48.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:13:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:13:48.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T13:13:48.243 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:13:48.231+0000 7f61b1855700 -1 reset not still connected to 0x557b229664e0 2024-09-16T13:13:48.243 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:13:48.231+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T13:13:48.243 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:13:48.231+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T13:13:48.243 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:13:48.231+0000 7f61b1855700 -1 reset not still connected to 0x557b3b101c70 2024-09-16T13:13:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:13:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 46c5f7ce-56e5-4725-9f6f-d1c94f8c9c3e 2024-09-16T13:13:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:13:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63909888.0,"last_snapshot_bytes":63909888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"} 2024-09-16T13:13:48.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:13:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:13:23 2024-09-16T13:13:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:13:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:13:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:13:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:13:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:13:43' 2024-09-16T13:13:48.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:13:48.308 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:13:48.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:13:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63909888.0,"last_snapshot_bytes":63909888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"} 2024-09-16T13:13:48.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:13:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:13:48.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:13:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:13:48.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:13:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:13:48.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:13:48.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:13:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:13:23' 2024-09-16T13:13:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-16 13:13:23' 2024-09-16T13:13:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:13:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:48.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:13:23' 2024-09-16T13:13:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:13:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:13:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:13:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK742544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70103040.0,"last_snapshot_bytes":70103040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:43image-primary446c5f7ce-56e5-4725-9f6f-d1c94f8c9c3eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63909888.0,"last_snapshot_bytes":63909888,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:43image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":114204672.0,"last_snapshot_bytes":114204672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:43image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65126400.0,"last_snapshot_bytes":65126400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:43image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62160896.0,"last_snapshot_bytes":62160896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:43image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:43image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:43' 2024-09-16T13:13:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:13:48.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T13:13:48.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:13:23' 2024-09-16T13:13:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T13:13:48.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:13:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:13:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:13:48.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:13:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:48.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-16T13:13:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:13:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:13:48.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-16T13:13:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-16T13:13:50.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:13:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-16T13:13:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:13:50.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:13:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary4 2024-09-16T13:13:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:13:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary4 2024-09-16T13:13:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:13:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary5 2024-09-16T13:13:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-16T13:13:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:13:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:13:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:13:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:13:50.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:13:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-16T13:13:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:51.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:13:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:13:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:13:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:13:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:13:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T13:13:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:13:51.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 40484d62-c2e4-4d70-a7a6-1d77d8ae20f2 2024-09-16T13:13:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:13:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":114204672.0,"last_snapshot_bytes":114204672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"} 2024-09-16T13:13:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:13:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:13:23 2024-09-16T13:13:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:13:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:13:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:13:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:13:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:13:48' 2024-09-16T13:13:51.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:13:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:13:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:13:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":114204672.0,"last_snapshot_bytes":114204672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"} 2024-09-16T13:13:51.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:13:51.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:13:51.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:13:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:13:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:13:51.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:13:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:13:51.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:13:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:13:23' 2024-09-16T13:13:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-16 13:13:23' 2024-09-16T13:13:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:13:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:13:23' 2024-09-16T13:13:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:13:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:13:51.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:13:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK642544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70103040.0,"last_snapshot_bytes":70103040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:48image-primary540484d62-c2e4-4d70-a7a6-1d77d8ae20f2up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":114204672.0,"last_snapshot_bytes":114204672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:48image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65126400.0,"last_snapshot_bytes":65126400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:48image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62160896.0,"last_snapshot_bytes":62160896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:48image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:48image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:48' 2024-09-16T13:13:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:13:51.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T13:13:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:13:23' 2024-09-16T13:13:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T13:13:51.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:13:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:13:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:13:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:13:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-16T13:13:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:13:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:13:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-16T13:13:51.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-16T13:13:53.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:13:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-16T13:13:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:53.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:13:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:13:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:13:53.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary5 2024-09-16T13:13:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:53.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary5 2024-09-16T13:13:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:13:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary6 2024-09-16T13:13:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-16T13:13:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:13:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:13:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:13:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:13:53.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:13:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-16T13:13:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:13:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:54.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:13:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:13:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:13:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:13:54.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T13:13:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:13:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 3483ebd3-b9b7-4f5f-86db-00f691453f56 2024-09-16T13:13:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:13:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65126400.0,"last_snapshot_bytes":65126400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"} 2024-09-16T13:13:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:13:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:13:53 2024-09-16T13:13:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:13:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:13:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:13:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:13:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:13:53' 2024-09-16T13:13:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:13:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:13:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:13:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65126400.0,"last_snapshot_bytes":65126400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"} 2024-09-16T13:13:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:13:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:13:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:13:54.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:13:54.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:13:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:13:54.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:13:54.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:13:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:13:53' 2024-09-16T13:13:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-16 13:13:53' 2024-09-16T13:13:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:13:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:13:53' 2024-09-16T13:13:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:13:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:13:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:13:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK542544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70103040.0,"last_snapshot_bytes":70103040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:53image-primary63483ebd3-b9b7-4f5f-86db-00f691453f56up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65126400.0,"last_snapshot_bytes":65126400,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:53image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62160896.0,"last_snapshot_bytes":62160896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:53image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:53image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:53' 2024-09-16T13:13:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:13:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T13:13:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:13:53' 2024-09-16T13:13:54.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T13:13:54.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:13:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:13:54.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:13:54.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:13:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-16T13:13:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:13:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:13:54.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-16T13:13:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-16T13:13:56.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:13:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-16T13:13:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:13:56.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:13:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:13:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary6 2024-09-16T13:13:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary6 2024-09-16T13:13:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:13:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary7 2024-09-16T13:13:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-16T13:13:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:13:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:13:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:13:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:13:56.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:13:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-16T13:13:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:13:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:13:57.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:13:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:13:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:13:57.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T13:13:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:13:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 270aa00f-3fdb-4e45-9116-ea259ca6b320 2024-09-16T13:13:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:13:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62160896.0,"last_snapshot_bytes":62160896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"} 2024-09-16T13:13:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:13:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:13:53 2024-09-16T13:13:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:13:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:13:57.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:13:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:13:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:13:53' 2024-09-16T13:13:57.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:13:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:13:57.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:13:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62160896.0,"last_snapshot_bytes":62160896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"} 2024-09-16T13:13:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:13:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:13:57.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:13:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:13:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:13:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:13:57.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:13:57.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:13:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:13:53' 2024-09-16T13:13:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-16 13:13:53' 2024-09-16T13:13:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:13:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:13:53' 2024-09-16T13:13:57.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:13:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:13:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:13:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK442544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70103040.0,"last_snapshot_bytes":70103040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:53image-primary7270aa00f-3fdb-4e45-9116-ea259ca6b320up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62160896.0,"last_snapshot_bytes":62160896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:53image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:53image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:53' 2024-09-16T13:13:57.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:13:57.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T13:13:57.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:13:53' 2024-09-16T13:13:57.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T13:13:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:13:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:13:57.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:13:57.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:13:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-16T13:13:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:13:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:13:57.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-16T13:13:57.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-16T13:13:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:13:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-16T13:13:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:13:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:13:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:13:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary7 2024-09-16T13:13:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:13:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary7 2024-09-16T13:13:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:13:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary8 2024-09-16T13:13:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-16T13:13:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:13:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:13:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:13:59.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:13:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:13:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:13:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:13:59.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:13:59.783 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:13:59.771+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1748f0 2024-09-16T13:13:59.791 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:13:59.779+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174d00 2024-09-16T13:14:00.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-16T13:14:00.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:14:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:14:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:14:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:14:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:14:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:14:00.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T13:14:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:14:00.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: d2b71a04-786f-44dd-bc9a-3d01975d02f7 2024-09-16T13:14:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:14:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"} 2024-09-16T13:14:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:14:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:13:53 2024-09-16T13:14:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:14:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:14:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:14:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:14:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:13:58' 2024-09-16T13:14:00.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:14:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:14:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:14:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"} 2024-09-16T13:14:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:14:00.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:14:00.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:14:00.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:14:00.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:14:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:14:00.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:14:00.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:14:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:13:53' 2024-09-16T13:14:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-16 13:13:53' 2024-09-16T13:14:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:14:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:14:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:13:53' 2024-09-16T13:14:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:14:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:14:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:14:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK342544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70103040.0,"last_snapshot_bytes":70103040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:58image-primary8d2b71a04-786f-44dd-bc9a-3d01975d02f7up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:58image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:58' 2024-09-16T13:14:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:14:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T13:14:00.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:13:53' 2024-09-16T13:14:00.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T13:14:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:14:00.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:14:00.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:14:00.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:14:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:14:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:14:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-16T13:14:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:14:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:14:00.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-16T13:14:00.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-16T13:14:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:14:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-16T13:14:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:14:02.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:14:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:14:02.145 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary8 2024-09-16T13:14:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:14:02.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary8 2024-09-16T13:14:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:14:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary9 2024-09-16T13:14:02.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-16T13:14:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:14:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:14:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:14:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:14:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:14:02.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:14:02.959 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:14:02.947+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1748f0 2024-09-16T13:14:03.130 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:14:03.119+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1748f0 2024-09-16T13:14:03.130 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:14:03.119+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174d00 2024-09-16T13:14:03.137 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:14:03.127+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T13:14:03.137 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:14:03.127+0000 7ff4f6946700 -1 reset not still connected to 0x557557038ea0 2024-09-16T13:14:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-16T13:14:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:14:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:14:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:14:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:14:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:14:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:14:03.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T13:14:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:14:03.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 3041a084-27f4-42db-87ac-d71aabb63b76 2024-09-16T13:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"} 2024-09-16T13:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:13:53 2024-09-16T13:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:13:58' 2024-09-16T13:14:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:14:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:14:03.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:14:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"} 2024-09-16T13:14:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:14:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:14:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:14:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:14:03.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:14:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:14:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:14:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:14:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:13:53' 2024-09-16T13:14:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-16 13:13:53' 2024-09-16T13:14:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:14:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:14:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:13:53' 2024-09-16T13:14:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:14:03.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:14:03.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:14:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK242544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70103040.0,"last_snapshot_bytes":70103040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:58image-primary93041a084-27f4-42db-87ac-d71aabb63b76up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70004736.0,"last_snapshot_bytes":70004736,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:13:58' 2024-09-16T13:14:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:14:03.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T13:14:03.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:13:53' 2024-09-16T13:14:03.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T13:14:03.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:14:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:14:03.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:14:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:14:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:14:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:14:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-16T13:14:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:14:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:14:03.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-16T13:14:03.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-16T13:14:05.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:14:05.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-16T13:14:05.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:05.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:05.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:14:05.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:14:05.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:14:05.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary9 2024-09-16T13:14:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:14:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary9 2024-09-16T13:14:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:14:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary10 2024-09-16T13:14:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-16T13:14:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:14:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:14:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:14:05.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:14:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:14:05.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:14:06.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-16T13:14:06.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:06.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:06.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:14:06.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:14:06.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:14:06.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:14:06.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:14:06.142 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:14:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T13:14:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:14:06.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b6a5803f-d8e8-4bd6-8abc-6f43c1584d8a 2024-09-16T13:14:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:14:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70103040.0,"last_snapshot_bytes":70103040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"} 2024-09-16T13:14:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:14:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:13:53 2024-09-16T13:14:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:14:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:14:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:14:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:14:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:14:03' 2024-09-16T13:14:06.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:14:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:14:06.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:14:06.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70103040.0,"last_snapshot_bytes":70103040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"} 2024-09-16T13:14:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:14:06.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:14:06.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:14:06.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:14:06.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:14:06.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:14:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:14:06.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:14:06.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:13:53' 2024-09-16T13:14:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-16 13:13:53' 2024-09-16T13:14:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:14:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:14:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:13:53' 2024-09-16T13:14:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:14:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:14:06.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:14:06.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK142544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10b6a5803f-d8e8-4bd6-8abc-6f43c1584d8aup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70103040.0,"last_snapshot_bytes":70103040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492318,"remote_snapshot_timestamp":1726492318,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:13:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:14:03' 2024-09-16T13:14:06.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:14:06.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T13:14:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:13:53' 2024-09-16T13:14:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T13:14:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:14:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:14:06.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:14:06.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:14:06.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:14:06.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:14:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-16T13:14:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:14:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:14:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-16T13:14:06.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-16T13:14:08.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:14:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-16T13:14:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:14:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:14:08.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:14:08.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary10 2024-09-16T13:14:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:14:08.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary10 2024-09-16T13:14:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..10} 2024-09-16T13:14:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS=() 2024-09-16T13:14:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS=() 2024-09-16T13:14:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS=() 2024-09-16T13:14:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS=() 2024-09-16T13:14:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ RET=0 2024-09-16T13:14:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:14:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary1 2024-09-16T13:14:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary1 2024-09-16T13:14:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-16T13:14:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:08.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:14:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:14:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:14:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-16T13:14:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:08.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:14:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:14:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:14:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:14:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-16T13:14:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:14:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-16T13:14:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:14:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:14:08.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-16T13:14:09.143 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:14:09.131+0000 7f61b1855700 -1 reset not still connected to 0x557b1f47fd40 2024-09-16T13:14:09.143 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:14:09.131+0000 7f61b1855700 -1 reset not still connected to 0x557b229664e0 2024-09-16T13:14:09.143 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:14:09.131+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T13:14:09.143 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:14:09.131+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T13:14:09.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:14:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-16T13:14:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary1': 2024-09-16T13:14:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:14:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:14:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:14:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 2c83232d29f1 2024-09-16T13:14:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.2c83232d29f1 2024-09-16T13:14:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:14:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:14:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:14:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:14:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:14:08 2024 2024-09-16T13:14:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:14:08 2024 2024-09-16T13:14:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:14:08 2024 2024-09-16T13:14:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:14:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:14:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 2c59192a-bbce-40d9-8377-cfe3d85c4f4d 2024-09-16T13:14:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:14:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:14:09.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary1 2024-09-16T13:14:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd0 2024-09-16T13:14:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:14:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd0 2024-09-16T13:14:09.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:14:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:14:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:14:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: f262ce87-ca89-4af5-a669-ee54d2ad2939 2024-09-16T13:14:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:14:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:14:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:14:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:14:09.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:14:10.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:14:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:14:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:14:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary1 2024-09-16T13:14:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd0 test-primary1 2024-09-16T13:14:10.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:14:10.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary1 2024-09-16T13:14:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:14:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-16T13:14:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:14:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:14:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:14:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:14:10.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:14:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:14:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary2 2024-09-16T13:14:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary2 2024-09-16T13:14:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-16T13:14:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary1 2024-09-16T13:14:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:14:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:14:10.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:14:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-16T13:14:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary1 2024-09-16T13:14:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:14:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-16T13:14:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:14:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:14:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:14:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-16T13:14:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:14:10.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-16T13:14:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:14:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:14:10.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-16T13:14:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:14:10.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-16T13:14:11.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3958 2024-09-16T13:14:11.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:11.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:14:11.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-16T13:14:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary2': 2024-09-16T13:14:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:14:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:14:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:14:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 2c9152168470 2024-09-16T13:14:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.2c9152168470 2024-09-16T13:14:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:14:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:14:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:14:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:14:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:14:10 2024 2024-09-16T13:14:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:14:10 2024 2024-09-16T13:14:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:14:10 2024 2024-09-16T13:14:11.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:14:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:14:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156 2024-09-16T13:14:11.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:14:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:14:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary2 2024-09-16T13:14:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd1 2024-09-16T13:14:11.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:14:11.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd1 2024-09-16T13:14:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:14:11.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:14:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:14:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 264b0986-69f1-436d-9a94-ac88d0890c28 2024-09-16T13:14:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:14:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:14:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:14:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:14:11.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:14:12.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:14:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:14:12.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:14:12.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary2 2024-09-16T13:14:12.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd1 test-primary2 2024-09-16T13:14:12.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:14:12.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary2 2024-09-16T13:14:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:14:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-16T13:14:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:14:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:14:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:12.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:14:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:14:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:14:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary2 2024-09-16T13:14:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:14:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary3 2024-09-16T13:14:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary3 2024-09-16T13:14:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary2 2024-09-16T13:14:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-16T13:14:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-16T13:14:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:12.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:14:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:14:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:14:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-16T13:14:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:14:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:14:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:14:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:14:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:12.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-16T13:14:12.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:14:12.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-16T13:14:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:14:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-16T13:14:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:14:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:14:12.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-16T13:14:13.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3961 2024-09-16T13:14:13.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:14:13.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-16T13:14:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary3': 2024-09-16T13:14:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:14:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:14:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:14:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 2ca2578ade67 2024-09-16T13:14:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.2ca2578ade67 2024-09-16T13:14:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:14:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:14:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:14:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:14:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:14:12 2024 2024-09-16T13:14:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:14:12 2024 2024-09-16T13:14:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:14:12 2024 2024-09-16T13:14:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:14:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:14:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 503485e0-28c1-4dd2-9faa-ff8fcbad267f 2024-09-16T13:14:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:14:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:14:13.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary3 2024-09-16T13:14:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd2 2024-09-16T13:14:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:14:13.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd2 2024-09-16T13:14:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:14:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:14:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:14:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 657b01cf-8b5a-4536-9803-6be3c0ba2229 2024-09-16T13:14:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:14:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:14:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:14:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:14:13.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:14:14.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:14:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:14:14.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:14:14.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary3 2024-09-16T13:14:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd2 test-primary3 2024-09-16T13:14:14.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:14:14.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary3 2024-09-16T13:14:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:14:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-16T13:14:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:14:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:14.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:14:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:14:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:14:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:14:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:14:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary3 2024-09-16T13:14:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary4 2024-09-16T13:14:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary4 2024-09-16T13:14:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-16T13:14:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:14.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary3 2024-09-16T13:14:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-16T13:14:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:14:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:14:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:14:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:14.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-16T13:14:14.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:14:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:14:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:14:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:14:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-16T13:14:14.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:14:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:14:14.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:14:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:14:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-16T13:14:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:14:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:14:14.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-16T13:14:14.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:14:14.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-16T13:14:14.929 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:14:14.915+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1748f0 2024-09-16T13:14:14.930 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:14:14.915+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174d00 2024-09-16T13:14:14.930 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:14:14.919+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T13:14:14.930 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:14:14.919+0000 7ff4f6946700 -1 reset not still connected to 0x55754fed8dd0 2024-09-16T13:14:15.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3963 2024-09-16T13:14:15.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:14:15.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:15.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-16T13:14:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary4': 2024-09-16T13:14:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:14:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:14:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:14:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 2caf408ee5e8 2024-09-16T13:14:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.2caf408ee5e8 2024-09-16T13:14:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:14:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:14:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:14:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:14:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:14:14 2024 2024-09-16T13:14:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:14:14 2024 2024-09-16T13:14:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:14:14 2024 2024-09-16T13:14:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:14:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:14:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 6e9c1af9-867d-4cc5-b8a8-a0083566b5f4 2024-09-16T13:14:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:14:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:14:15.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary4 2024-09-16T13:14:15.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd3 2024-09-16T13:14:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:14:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd3 2024-09-16T13:14:15.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:14:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:14:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:14:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 31cfa861-c173-4400-ad36-46feede5d84e 2024-09-16T13:14:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:14:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:14:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:14:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:14:15.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:14:16.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:14:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:14:16.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:14:16.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary4 2024-09-16T13:14:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd3 test-primary4 2024-09-16T13:14:16.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:14:16.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary4 2024-09-16T13:14:16.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:14:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:14:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-16T13:14:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:14:16.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary5 2024-09-16T13:14:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary5 2024-09-16T13:14:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-16T13:14:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:14:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary4 2024-09-16T13:14:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:14:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:16.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:14:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:14:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:14:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:14:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-16T13:14:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:14:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:14:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary4 2024-09-16T13:14:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:14:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-16T13:14:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:14:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:14:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:16.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-16T13:14:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:14:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-16T13:14:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:14:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:14:16.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-16T13:14:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:14:16.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-16T13:14:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3964 2024-09-16T13:14:16.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:14:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:14:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:14:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3968 2024-09-16T13:14:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:17.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:14:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-16T13:14:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary5': 2024-09-16T13:14:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:14:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:14:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:14:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 2cc2a26733a4 2024-09-16T13:14:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.2cc2a26733a4 2024-09-16T13:14:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:14:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:14:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:14:17.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:14:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:14:16 2024 2024-09-16T13:14:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:14:16 2024 2024-09-16T13:14:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:14:16 2024 2024-09-16T13:14:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:14:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:14:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: a5f856a7-332e-4bbf-9058-5deb68a13aee 2024-09-16T13:14:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:14:17.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:14:17.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary5 2024-09-16T13:14:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd4 2024-09-16T13:14:17.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:14:17.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd4 2024-09-16T13:14:17.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:14:18.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/26214401576960/2621440 done 2024-09-16T13:14:18.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:14:18.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 6b248daf-87b3-4b76-a117-039ce40921f1 2024-09-16T13:14:18.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:14:18.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:14:18.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:14:18.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:14:18.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:14:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): Snapshot ID: 3971 2024-09-16T13:14:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:18.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T13:14:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:14:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:14:18.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:14:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:14:18.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:14:18.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary5 2024-09-16T13:14:18.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd4 test-primary5 2024-09-16T13:14:18.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:14:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary5 2024-09-16T13:14:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:14:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-16T13:14:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:18.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:14:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:14:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:14:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:14:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary5 2024-09-16T13:14:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:14:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary6 2024-09-16T13:14:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary6 2024-09-16T13:14:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:14:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-16T13:14:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary5 2024-09-16T13:14:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-16T13:14:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:18.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:14:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:14:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:14:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-16T13:14:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:14:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:14:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:14:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:14:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:18.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-16T13:14:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:14:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-16T13:14:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:14:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:14:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-16T13:14:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:14:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-16T13:14:18.837 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:14:18.826+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T13:14:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:14:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:14:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:14:19.457 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:14:19.446+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174d00 2024-09-16T13:14:19.458 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:14:19.446+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T13:14:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:14:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:19.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:14:19.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:14:19.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3975 2024-09-16T13:14:19.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:14:19.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:19.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-16T13:14:19.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary6': 2024-09-16T13:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 2cd8a531ff95 2024-09-16T13:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.2cd8a531ff95 2024-09-16T13:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:14:18 2024 2024-09-16T13:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:14:18 2024 2024-09-16T13:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:14:18 2024 2024-09-16T13:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:14:19.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:14:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 3c6cd560-e2c0-4db8-aea1-e5d13b858250 2024-09-16T13:14:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:14:19.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:14:19.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary6 2024-09-16T13:14:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd5 2024-09-16T13:14:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:14:19.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd5 2024-09-16T13:14:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:14:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:14:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:14:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: b1c9c2ca-a451-483a-bb0c-22e811957216 2024-09-16T13:14:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:14:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:14:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:14:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:14:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:14:20.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:14:20.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 Snapshot ID: 3974 2024-09-16T13:14:20.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T13:14:20.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:14:20.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary6 2024-09-16T13:14:20.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd5 test-primary6 2024-09-16T13:14:20.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:20.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:14:20.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary6 2024-09-16T13:14:20.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:14:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:14:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-16T13:14:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:14:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary7 2024-09-16T13:14:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary7 2024-09-16T13:14:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-16T13:14:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary6 2024-09-16T13:14:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:14:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:14:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:14:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:14:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-16T13:14:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:14:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:20.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:14:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary6 2024-09-16T13:14:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:14:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:14:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:14:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:14:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-16T13:14:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:14:20.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-16T13:14:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:14:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-16T13:14:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:14:20.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:14:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-16T13:14:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:14:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-16T13:14:21.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:14:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:14:21.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:14:21.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3979 2024-09-16T13:14:21.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:21.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3977 2024-09-16T13:14:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:21.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3982 2024-09-16T13:14:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:14:21.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:21.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-16T13:14:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary7': 2024-09-16T13:14:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:14:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:14:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:14:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 2cedbfd10d3f 2024-09-16T13:14:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.2cedbfd10d3f 2024-09-16T13:14:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:14:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:14:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:14:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:14:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:14:20 2024 2024-09-16T13:14:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:14:20 2024 2024-09-16T13:14:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:14:20 2024 2024-09-16T13:14:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:14:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:14:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: e78da270-e064-4749-9b3c-5b1059782eb5 2024-09-16T13:14:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:14:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:14:21.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary7 2024-09-16T13:14:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd6 2024-09-16T13:14:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:14:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd6 2024-09-16T13:14:21.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:14:21.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:14:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:14:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 88fb34cc-c3c1-4b36-bd2c-9cbc7c0b3bbb 2024-09-16T13:14:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:14:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:14:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:14:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:14:21.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:14:22.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:14:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:14:22.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:14:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary7 2024-09-16T13:14:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd6 test-primary7 2024-09-16T13:14:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:14:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary7 2024-09-16T13:14:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:14:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-16T13:14:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:14:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:14:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:14:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary8 2024-09-16T13:14:22.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary7 2024-09-16T13:14:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary8 2024-09-16T13:14:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:14:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-16T13:14:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:14:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:14:22.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:14:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary7 2024-09-16T13:14:22.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:14:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:14:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-16T13:14:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:22.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-16T13:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-16T13:14:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:14:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-16T13:14:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:22.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:22.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:14:22.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:14:22.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-16T13:14:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:14:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-16T13:14:22.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3986 2024-09-16T13:14:22.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:22.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:14:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:14:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:14:22.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:14:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-16T13:14:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary8': 2024-09-16T13:14:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:14:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:14:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:14:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 2d02c70e3091 2024-09-16T13:14:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.2d02c70e3091 2024-09-16T13:14:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:14:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:14:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:14:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:14:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:14:22 2024 2024-09-16T13:14:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:14:22 2024 2024-09-16T13:14:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:14:22 2024 2024-09-16T13:14:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:14:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:14:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 3b49bdc8-de30-4087-b019-4db5f1999408 2024-09-16T13:14:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:14:22.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:14:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary8 2024-09-16T13:14:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd7 2024-09-16T13:14:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:14:22.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd7 2024-09-16T13:14:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:14:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:14:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:14:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: df398430-d22b-4aef-82ff-6b469d886ddc 2024-09-16T13:14:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:14:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:14:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:14:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:14:22.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:14:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:14:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:14:23.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:14:23.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary8 2024-09-16T13:14:23.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd7 test-primary8 2024-09-16T13:14:23.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:14:23.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary8 2024-09-16T13:14:23.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:14:23.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-16T13:14:23.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:14:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:14:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:14:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:14:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:14:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:14:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary9 2024-09-16T13:14:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary9 2024-09-16T13:14:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-16T13:14:23.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:14:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary8 2024-09-16T13:14:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:14:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:14:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary8 2024-09-16T13:14:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-16T13:14:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-16T13:14:23.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:23.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:14:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:14:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:14:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:14:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-16T13:14:23.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3984 2024-09-16T13:14:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:14:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:23.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:14:23.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:14:23.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:14:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-16T13:14:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:14:23.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:14:23.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-16T13:14:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:14:23.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-16T13:14:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:14:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:14:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:14:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3990 2024-09-16T13:14:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:14:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:14:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:14:24.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:24.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3992 2024-09-16T13:14:24.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:14:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:14:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:14:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:14:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-16T13:14:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary9': 2024-09-16T13:14:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:14:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:14:24.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:14:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 2d15d5c820f0 2024-09-16T13:14:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.2d15d5c820f0 2024-09-16T13:14:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:14:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:14:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:14:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:14:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:14:23 2024 2024-09-16T13:14:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:14:23 2024 2024-09-16T13:14:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:14:23 2024 2024-09-16T13:14:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:14:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:14:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: d0955fd5-aa21-42b5-8d3d-c814e36c8838 2024-09-16T13:14:24.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:14:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:14:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary9 2024-09-16T13:14:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd8 2024-09-16T13:14:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:14:24.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd8 2024-09-16T13:14:24.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:14:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/26214402101248/2621440 done 2024-09-16T13:14:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:14:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 8308adb1-66b0-440d-b3f5-4394ea0f67a4 2024-09-16T13:14:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:14:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:14:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:14:25.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:14:25.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:14:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:14:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:25.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:14:25.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:14:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:14:25.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 Snapshot ID: 3993 2024-09-16T13:14:25.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T13:14:25.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:14:25.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary9 2024-09-16T13:14:25.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd8 test-primary9 2024-09-16T13:14:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:14:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary9 2024-09-16T13:14:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:14:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:14:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-16T13:14:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:14:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary10 2024-09-16T13:14:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary10 2024-09-16T13:14:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-16T13:14:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:14:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:14:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:14:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:14:25.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-16T13:14:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:14:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:14:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:14:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:14:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:14:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:14:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:14:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:14:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-16T13:14:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary9 2024-09-16T13:14:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary9 2024-09-16T13:14:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-16T13:14:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:14:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-16T13:14:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:14:25.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:14:25.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-16T13:14:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:14:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-16T13:14:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:14:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:14:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:14:26.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3997 2024-09-16T13:14:26.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3999 2024-09-16T13:14:26.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 3996 2024-09-16T13:14:26.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:26.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:26.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:26.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:14:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4004 2024-09-16T13:14:26.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-16T13:14:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary10': 2024-09-16T13:14:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:14:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:14:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:14:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 2d32cc3aa87a 2024-09-16T13:14:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.2d32cc3aa87a 2024-09-16T13:14:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:14:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:14:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:14:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:14:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:14:25 2024 2024-09-16T13:14:26.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:14:25 2024 2024-09-16T13:14:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:14:25 2024 2024-09-16T13:14:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:14:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:14:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: bf544298-988b-41f0-a03c-856c55772ccd 2024-09-16T13:14:26.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:14:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:14:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary10 2024-09-16T13:14:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd9 2024-09-16T13:14:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:14:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd9 2024-09-16T13:14:26.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:14:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/26214402101248/2621440 done 2024-09-16T13:14:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:14:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 76119e7c-6a1f-47fe-9643-9250da3b0b48 2024-09-16T13:14:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:14:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:14:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:14:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:14:27.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:14:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:14:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 Snapshot ID: 4000 2024-09-16T13:14:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:14:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:14:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:14:27.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:14:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:14:27.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:14:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T13:14:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:14:27.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary10 2024-09-16T13:14:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd9 test-primary10 2024-09-16T13:14:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:14:27.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary10 2024-09-16T13:14:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:14:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-16T13:14:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:14:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:14:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:14:27.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:14:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:14:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary10 2024-09-16T13:14:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:14:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 97968 2024-09-16T13:14:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary10 2024-09-16T13:14:27.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-16T13:14:27.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:14:27.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-16T13:14:28.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4006 2024-09-16T13:14:28.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4011 2024-09-16T13:14:28.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:14:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:14:28.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:14:29.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4010 2024-09-16T13:14:29.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4008 2024-09-16T13:14:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:14:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:14:29.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:14:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:14:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:14:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:14:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:14:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:14:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:14:29.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:14:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:14:29.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:14:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:14:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:14:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:14:30.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4015 2024-09-16T13:14:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:30.646 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:14:30.638+0000 7f61b1855700 -1 reset not still connected to 0x557b43e47ee0 2024-09-16T13:14:30.646 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:14:30.638+0000 7f61b1855700 -1 reset not still connected to 0x557b4b50e9c0 2024-09-16T13:14:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:14:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:14:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:14:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:14:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:14:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:14:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4019 2024-09-16T13:14:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4018 2024-09-16T13:14:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:31.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4020 2024-09-16T13:14:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4021 2024-09-16T13:14:31.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:31.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:31.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:32.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4024 2024-09-16T13:14:32.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:32.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:14:32.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:32.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:32.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:14:32.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:14:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:14:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:14:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:14:32.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:33.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4029 2024-09-16T13:14:33.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4028 2024-09-16T13:14:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:14:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:14:33.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:14:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4032 2024-09-16T13:14:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4031 2024-09-16T13:14:34.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:14:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:14:34.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:14:34.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:14:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:14:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:14:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:14:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:14:34.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:14:34.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:14:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:14:34.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:14:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4035 2024-09-16T13:14:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:14:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:14:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:14:36.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4038 2024-09-16T13:14:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:14:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:14:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:14:36.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:36.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4041 2024-09-16T13:14:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:14:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:14:36.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:14:36.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4042 2024-09-16T13:14:36.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4040 2024-09-16T13:14:36.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4047 2024-09-16T13:14:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:14:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:37.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:14:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:14:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:14:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:14:37.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:14:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4049 2024-09-16T13:14:38.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:38.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4050 2024-09-16T13:14:38.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:14:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:14:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:14:39.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4054 2024-09-16T13:14:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4053 2024-09-16T13:14:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:39.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:14:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:14:39.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:14:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:14:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:14:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:14:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:14:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:14:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:14:39.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:14:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:14:39.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:14:40.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4057 2024-09-16T13:14:40.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:14:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:40.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:14:40.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:14:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:14:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:14:41.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:14:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4059 2024-09-16T13:14:41.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4060 2024-09-16T13:14:41.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:41.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:14:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:14:41.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:14:41.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:41.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4061 2024-09-16T13:14:41.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4062 2024-09-16T13:14:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:41.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:42.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4067 2024-09-16T13:14:42.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:14:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:42.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:14:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:14:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:14:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:14:42.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:14:43.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4070 2024-09-16T13:14:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4069 2024-09-16T13:14:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:14:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:14:43.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:14:43.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:44.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4074 2024-09-16T13:14:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4072 2024-09-16T13:14:44.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:44.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:14:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:14:44.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:14:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:14:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:14:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:14:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:14:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:14:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:14:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:14:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:14:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:14:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:14:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:14:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:14:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4077 2024-09-16T13:14:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4082 2024-09-16T13:14:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:14:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:46.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:14:46.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:14:46.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4079 2024-09-16T13:14:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:14:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:14:46.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:14:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4080 2024-09-16T13:14:46.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:46.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4081 2024-09-16T13:14:46.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:46.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:47.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:47.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:14:47.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:47.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:47.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:14:47.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:14:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:14:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:47.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:14:47.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:14:47.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4085 2024-09-16T13:14:47.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4090 2024-09-16T13:14:48.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4088 2024-09-16T13:14:48.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:48.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:14:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:14:48.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:14:49.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4092 2024-09-16T13:14:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4094 2024-09-16T13:14:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:14:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:14:49.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:14:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:49.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:14:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:14:49.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:14:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:14:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:14:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:14:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:14:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:14:49.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:14:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:14:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:14:50.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:14:50.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4097 2024-09-16T13:14:50.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:51.084 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:14:51.078+0000 7f61b1855700 -1 reset not still connected to 0x557b22966000 2024-09-16T13:14:51.088 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:14:51.082+0000 7f61b1855700 -1 reset not still connected to 0x557b229665b0 2024-09-16T13:14:51.090 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:14:51.082+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0ea0 2024-09-16T13:14:51.090 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:14:51.082+0000 7f61b1855700 -1 reset not still connected to 0x557b22af11e0 2024-09-16T13:14:51.091 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:14:51.082+0000 7f61b1855700 -1 reset not still connected to 0x557b317495f0 2024-09-16T13:14:51.091 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:14:51.082+0000 7f61b1855700 -1 reset not still connected to 0x557b32db05b0 2024-09-16T13:14:51.091 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:14:51.086+0000 7f61b1855700 -1 reset not still connected to 0x557b43e47ee0 2024-09-16T13:14:51.091 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:14:51.086+0000 7f61b1855700 -1 reset not still connected to 0x557b45cffd40 2024-09-16T13:14:51.091 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:14:51.086+0000 7f61b1855700 -1 reset not still connected to 0x557b4b50e9c0 2024-09-16T13:14:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:14:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:14:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:14:51.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:14:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:14:51.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:14:51.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4102 2024-09-16T13:14:51.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4100 2024-09-16T13:14:51.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4099 2024-09-16T13:14:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:51.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:51.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4101 2024-09-16T13:14:51.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:52.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4106 2024-09-16T13:14:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:14:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:52.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:14:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:14:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:14:52.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:14:52.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:14:52.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:53.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4109 2024-09-16T13:14:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:14:53.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:14:53.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:14:53.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4108 2024-09-16T13:14:53.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:14:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:14:54.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:14:54.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4111 2024-09-16T13:14:54.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4112 2024-09-16T13:14:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:14:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:54.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:14:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:14:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:14:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:14:54.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:14:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:54.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:54.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:54.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:14:54.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:54.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:54.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:14:54.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:14:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:14:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:14:55.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:14:55.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4116 2024-09-16T13:14:55.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:56.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:14:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:14:56.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:14:56.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4122 2024-09-16T13:14:56.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4120 2024-09-16T13:14:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4121 2024-09-16T13:14:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:14:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:14:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:14:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4118 2024-09-16T13:14:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:56.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:56.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:57.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4124 2024-09-16T13:14:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:14:57.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:14:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:14:57.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:57.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:57.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:14:57.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:14:57.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:14:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4128 2024-09-16T13:14:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4129 2024-09-16T13:14:58.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:58.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:14:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:14:58.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:14:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:14:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:14:59.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:14:59.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4131 2024-09-16T13:14:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:59.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:14:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:14:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:14:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:14:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:14:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:14:59.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4132 2024-09-16T13:14:59.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:14:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:14:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:14:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:14:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:14:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:14:59.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:14:59.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:15:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:15:00.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:15:00.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4137 2024-09-16T13:15:00.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:15:01.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:15:01.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:15:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:01.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:15:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:15:01.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:15:01.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4141 2024-09-16T13:15:01.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:01.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4139 2024-09-16T13:15:01.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4140 2024-09-16T13:15:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4142 2024-09-16T13:15:01.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:01.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:02.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4143 2024-09-16T13:15:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:15:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:15:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:15:02.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:02.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:15:02.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:02.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:02.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:15:02.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:15:03.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4148 2024-09-16T13:15:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:15:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:15:03.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:15:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4149 2024-09-16T13:15:03.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:03.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4151 2024-09-16T13:15:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:15:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:15:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:15:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4152 2024-09-16T13:15:04.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:04.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:04.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:04.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:15:04.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:04.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:04.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:15:04.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:15:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:15:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:15:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:15:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:15:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:15:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:15:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:05.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:15:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:15:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:15:05.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4157 2024-09-16T13:15:05.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4158 2024-09-16T13:15:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:15:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:15:06.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:15:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:15:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:15:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:15:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4160 2024-09-16T13:15:06.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4162 2024-09-16T13:15:06.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4161 2024-09-16T13:15:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:06.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4164 2024-09-16T13:15:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:15:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:15:07.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:15:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:15:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:15:07.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:15:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:08.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4169 2024-09-16T13:15:08.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4168 2024-09-16T13:15:08.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:08.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:08.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:15:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:15:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:15:09.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4173 2024-09-16T13:15:09.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:09.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4174 2024-09-16T13:15:09.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:15:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:15:09.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:15:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:15:09.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:15:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:15:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:15:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:15:09.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:15:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:15:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:15:09.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:15:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4177 2024-09-16T13:15:10.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:15:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:15:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:15:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4179 2024-09-16T13:15:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:15:11.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:15:11.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:15:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4180 2024-09-16T13:15:11.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:11.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:15:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:15:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:15:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4182 2024-09-16T13:15:11.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:11.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4181 2024-09-16T13:15:11.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:15:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:12.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:15:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:15:12.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4187 2024-09-16T13:15:12.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:15:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:15:12.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:15:13.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:13.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:15:13.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:13.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:13.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:15:13.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:15:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4188 2024-09-16T13:15:13.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4189 2024-09-16T13:15:13.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:13.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4191 2024-09-16T13:15:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:15:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:15:14.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:15:14.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4194 2024-09-16T13:15:14.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:14.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:15:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:15:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:15:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:15:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:15:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:15:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:15:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:15:14.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:15:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:15:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:15.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:15:15.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:15:15.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4196 2024-09-16T13:15:15.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:16.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4199 2024-09-16T13:15:16.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:15:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:15:16.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:15:16.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:15:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:15:16.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:15:16.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4201 2024-09-16T13:15:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4202 2024-09-16T13:15:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4200 2024-09-16T13:15:16.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:16.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:17.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4205 2024-09-16T13:15:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:15:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:15:17.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:15:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:15:17.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:15:17.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:15:17.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:18.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4208 2024-09-16T13:15:18.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4209 2024-09-16T13:15:18.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:18.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:18.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:15:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:15:18.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:15:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:15:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:15:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:15:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4213 2024-09-16T13:15:19.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4214 2024-09-16T13:15:19.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:15:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:15:19.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:15:19.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:15:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:15:19.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:15:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:15:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:19.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:15:19.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:15:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:15:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:20.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:20.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:15:20.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:15:20.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4217 2024-09-16T13:15:20.412 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:15:20.406+0000 7f61b1855700 -1 reset not still connected to 0x557b32db05b0 2024-09-16T13:15:20.415 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:15:20.406+0000 7f61b1855700 -1 reset not still connected to 0x557b43e47ee0 2024-09-16T13:15:20.415 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:15:20.406+0000 7f61b1855700 -1 reset not still connected to 0x557b45cffd40 2024-09-16T13:15:20.415 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:15:20.406+0000 7f61b1855700 -1 reset not still connected to 0x557b4b50e9c0 2024-09-16T13:15:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:21.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:15:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:15:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:15:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:15:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:15:21.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:15:21.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4219 2024-09-16T13:15:21.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:21.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4222 2024-09-16T13:15:21.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4220 2024-09-16T13:15:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4221 2024-09-16T13:15:21.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:21.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:15:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:15:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:15:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:15:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:15:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:15:22.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4227 2024-09-16T13:15:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:15:23.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:15:23.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:15:23.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4228 2024-09-16T13:15:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4229 2024-09-16T13:15:23.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4232 2024-09-16T13:15:24.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:15:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:15:24.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:15:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4233 2024-09-16T13:15:24.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:24.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:15:24.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:24.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:24.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:15:24.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:15:24.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:15:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:15:24.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:15:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:15:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:15:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:15:25.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:25.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:15:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:15:25.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:15:25.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4237 2024-09-16T13:15:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:15:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:15:26.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:15:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:15:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:15:26.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:15:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4242 2024-09-16T13:15:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4238 2024-09-16T13:15:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4240 2024-09-16T13:15:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4241 2024-09-16T13:15:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:26.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:27.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:15:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:15:27.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:15:27.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4246 2024-09-16T13:15:27.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:15:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:15:27.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:15:28.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4248 2024-09-16T13:15:28.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4249 2024-09-16T13:15:28.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:28.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:15:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:15:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:15:29.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4254 2024-09-16T13:15:29.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4251 2024-09-16T13:15:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:15:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:15:29.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:15:29.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:15:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:15:29.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:15:29.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:15:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:15:29.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:15:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:15:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:15:29.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:15:30.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:15:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:15:30.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:15:30.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4257 2024-09-16T13:15:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:15:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:15:31.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:15:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:15:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:15:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:15:31.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4260 2024-09-16T13:15:31.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4259 2024-09-16T13:15:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4262 2024-09-16T13:15:31.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:31.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4261 2024-09-16T13:15:31.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:31.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:15:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:15:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:15:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:15:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:15:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:15:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4265 2024-09-16T13:15:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:33.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4268 2024-09-16T13:15:33.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4269 2024-09-16T13:15:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:33.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:15:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:15:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:15:34.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4273 2024-09-16T13:15:34.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4274 2024-09-16T13:15:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:15:34.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:15:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:15:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:15:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:15:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:15:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:15:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:15:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:15:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:15:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:34.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:15:34.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:15:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:15:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:15:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:15:35.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4277 2024-09-16T13:15:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:15:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:15:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:15:36.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4281 2024-09-16T13:15:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:15:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:15:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:15:36.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4279 2024-09-16T13:15:36.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4282 2024-09-16T13:15:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4280 2024-09-16T13:15:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:36.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:36.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:37.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4284 2024-09-16T13:15:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:15:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:15:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:15:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:15:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:15:37.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:15:37.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4289 2024-09-16T13:15:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4288 2024-09-16T13:15:38.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:38.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:15:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:15:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:15:39.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4293 2024-09-16T13:15:39.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4294 2024-09-16T13:15:39.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:39.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:15:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:15:39.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:15:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:15:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:15:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:15:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:15:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:15:39.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:15:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:15:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:15:39.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:15:40.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4297 2024-09-16T13:15:40.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:15:40.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:15:40.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:15:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:15:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:41.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:15:41.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:15:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:15:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:15:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:15:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4302 2024-09-16T13:15:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4299 2024-09-16T13:15:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4301 2024-09-16T13:15:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4300 2024-09-16T13:15:41.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:41.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:41.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:42.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4306 2024-09-16T13:15:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:15:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:15:42.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:15:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:42.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:15:42.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:42.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:42.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:15:42.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:15:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:15:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:15:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:15:43.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4308 2024-09-16T13:15:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4310 2024-09-16T13:15:43.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:43.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:44.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4314 2024-09-16T13:15:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4313 2024-09-16T13:15:44.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:15:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:44.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:15:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:15:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:15:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:15:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:15:44.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:15:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:15:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:15:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:15:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:15:44.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:15:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:15:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:15:45.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:15:45.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4317 2024-09-16T13:15:45.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4319 2024-09-16T13:15:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:15:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:15:46.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:15:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:15:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:15:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:15:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:46.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4321 2024-09-16T13:15:46.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4322 2024-09-16T13:15:46.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:46.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:46.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4320 2024-09-16T13:15:46.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:47.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:15:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:47.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:15:47.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:15:47.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4323 2024-09-16T13:15:47.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:15:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:15:47.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:15:47.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4328 2024-09-16T13:15:48.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4329 2024-09-16T13:15:48.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:48.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:48.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:15:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:15:48.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:15:49.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4333 2024-09-16T13:15:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:15:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:15:49.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:15:49.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4334 2024-09-16T13:15:49.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:49.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:15:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:15:49.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:15:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:15:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:49.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:15:49.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:15:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:15:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:15:49.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:15:50.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4337 2024-09-16T13:15:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:15:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:15:50.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:15:50.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:51.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:15:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:15:51.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:15:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:15:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:51.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:51.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:15:51.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:15:51.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4339 2024-09-16T13:15:51.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4340 2024-09-16T13:15:51.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:51.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4342 2024-09-16T13:15:51.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4341 2024-09-16T13:15:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:51.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:51.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:52.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4347 2024-09-16T13:15:52.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:15:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:15:52.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:15:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:15:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:15:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:15:53.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:15:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:15:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:15:53.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4349 2024-09-16T13:15:53.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4348 2024-09-16T13:15:53.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:53.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:54.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4353 2024-09-16T13:15:54.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:54.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4354 2024-09-16T13:15:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:15:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:15:54.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:15:54.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:15:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:15:54.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:15:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:15:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:15:54.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:15:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:15:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:15:54.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:15:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:15:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:15:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:15:55.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4357 2024-09-16T13:15:55.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:15:56.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:15:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:15:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:56.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:15:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:15:56.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:15:56.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4360 2024-09-16T13:15:56.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:56.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4359 2024-09-16T13:15:56.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4362 2024-09-16T13:15:56.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:56.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4361 2024-09-16T13:15:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:56.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:57.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4365 2024-09-16T13:15:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:15:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:15:57.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:15:57.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:15:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:15:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:15:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4369 2024-09-16T13:15:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4368 2024-09-16T13:15:58.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:58.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:15:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:15:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:15:59.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4373 2024-09-16T13:15:59.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:15:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:15:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:15:59.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4374 2024-09-16T13:15:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:15:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:15:59.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:15:59.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:15:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:15:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:15:59.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:15:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:15:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:15:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:15:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:15:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:15:59.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:16:00.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:00.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:16:00.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:00.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:00.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:16:00.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:16:00.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4377 2024-09-16T13:16:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:16:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:01.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:16:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:16:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:16:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:16:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:16:01.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4379 2024-09-16T13:16:01.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4382 2024-09-16T13:16:01.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4380 2024-09-16T13:16:01.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:01.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4381 2024-09-16T13:16:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4386 2024-09-16T13:16:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:16:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:02.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:16:02.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:16:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:02.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:16:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:16:02.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:16:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:16:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:16:03.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:16:03.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4389 2024-09-16T13:16:03.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4388 2024-09-16T13:16:03.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:03.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:04.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4393 2024-09-16T13:16:04.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4394 2024-09-16T13:16:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:16:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:16:04.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:16:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:16:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:16:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:16:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:16:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:16:04.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:16:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:16:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:16:04.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:16:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:16:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:16:05.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:16:05.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4397 2024-09-16T13:16:05.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4399 2024-09-16T13:16:06.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:16:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:16:06.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:16:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4400 2024-09-16T13:16:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:16:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:16:06.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:16:06.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4401 2024-09-16T13:16:06.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4402 2024-09-16T13:16:06.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:07.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:16:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:16:07.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:16:07.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4406 2024-09-16T13:16:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:16:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:07.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:16:07.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:16:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:08.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4408 2024-09-16T13:16:08.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4410 2024-09-16T13:16:08.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:08.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:08.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:08.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:16:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:16:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:16:09.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4411 2024-09-16T13:16:09.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:09.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4414 2024-09-16T13:16:09.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:16:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:16:09.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:16:09.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:16:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:16:09.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:16:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:16:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:16:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:16:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:16:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:16:09.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:16:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4417 2024-09-16T13:16:10.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:16:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:16:10.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:16:11.156 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:16:11.137+0000 7f61b1855700 -1 reset not still connected to 0x557b43e47ee0 2024-09-16T13:16:11.156 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:16:11.141+0000 7f61b1855700 -1 reset not still connected to 0x557b4b50e9c0 2024-09-16T13:16:11.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4420 2024-09-16T13:16:11.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4419 2024-09-16T13:16:11.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4422 2024-09-16T13:16:11.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4421 2024-09-16T13:16:11.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:16:11.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:16:11.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:16:11.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:11.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:16:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:16:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:16:11.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4427 2024-09-16T13:16:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:16:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:16:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:16:12.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:16:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:16:12.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:16:13.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:16:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:16:13.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:16:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4429 2024-09-16T13:16:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:13.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4430 2024-09-16T13:16:13.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:14.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:16:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:16:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:16:14.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4433 2024-09-16T13:16:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:16:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:14.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:16:14.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:16:14.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:16:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:16:14.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:16:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:16:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:16:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:16:14.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4434 2024-09-16T13:16:14.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:16:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:16:15.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:16:15.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4436 2024-09-16T13:16:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:16:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:16:16.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:16:16.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4439 2024-09-16T13:16:16.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4438 2024-09-16T13:16:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:16:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:16:16.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:16:16.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4441 2024-09-16T13:16:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4442 2024-09-16T13:16:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:16.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:16:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:16:17.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:16:17.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4443 2024-09-16T13:16:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:16:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:16:17.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:16:17.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4450 2024-09-16T13:16:18.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4448 2024-09-16T13:16:18.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:18.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:16:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:16:18.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:16:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4453 2024-09-16T13:16:19.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4454 2024-09-16T13:16:19.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:16:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:16:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:16:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:19.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:16:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:16:19.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:16:19.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:16:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:16:19.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:16:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:16:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:16:19.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:16:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4457 2024-09-16T13:16:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:16:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:16:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:16:20.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:21.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:16:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:16:21.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:16:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:16:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:16:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:16:21.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4459 2024-09-16T13:16:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4462 2024-09-16T13:16:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4460 2024-09-16T13:16:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4461 2024-09-16T13:16:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:21.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:21.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:22.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4467 2024-09-16T13:16:22.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:22.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:16:22.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:22.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:22.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:16:22.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:16:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:16:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:16:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:16:22.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:23.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4468 2024-09-16T13:16:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:16:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:16:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:16:23.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4469 2024-09-16T13:16:23.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:24.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4474 2024-09-16T13:16:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:16:24.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:16:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:16:24.466 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T13:16:24.457+0000 7f4071c2f700 -1 reset not still connected to 0x5635b89dc270 2024-09-16T13:16:24.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4473 2024-09-16T13:16:24.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:24.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:16:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:16:24.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:16:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:16:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:16:24.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:16:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:24.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:16:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:16:24.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:16:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:16:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:16:25.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:16:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4476 2024-09-16T13:16:25.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:16:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:16:26.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:16:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:16:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:16:26.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:16:26.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4479 2024-09-16T13:16:26.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4480 2024-09-16T13:16:26.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4482 2024-09-16T13:16:26.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:26.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:26.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4481 2024-09-16T13:16:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:27.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4486 2024-09-16T13:16:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:16:27.476 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:16:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:16:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:16:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:16:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:16:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:28.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4488 2024-09-16T13:16:28.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4489 2024-09-16T13:16:28.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:28.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:16:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:16:28.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:16:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:16:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:16:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:16:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4494 2024-09-16T13:16:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:16:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:16:29.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:16:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4493 2024-09-16T13:16:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:16:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:29.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:16:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:16:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:16:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:29.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:16:29.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:16:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:16:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:16:30.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:16:30.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4497 2024-09-16T13:16:30.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:16:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:16:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:16:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:16:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:16:31.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:16:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4501 2024-09-16T13:16:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4499 2024-09-16T13:16:31.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4500 2024-09-16T13:16:31.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4502 2024-09-16T13:16:31.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4505 2024-09-16T13:16:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:32.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:16:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:16:32.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:16:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:16:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:16:32.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:16:33.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4508 2024-09-16T13:16:33.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4509 2024-09-16T13:16:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:16:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:16:33.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:16:33.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:33.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:34.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4513 2024-09-16T13:16:34.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4514 2024-09-16T13:16:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:16:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:16:34.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:16:34.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:34.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:16:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:16:34.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:16:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:34.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:16:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:16:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:16:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:16:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:16:34.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:16:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:16:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:35.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:16:35.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:16:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4517 2024-09-16T13:16:35.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:36.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:16:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:16:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:16:36.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4520 2024-09-16T13:16:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:16:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:16:36.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:16:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4521 2024-09-16T13:16:36.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4522 2024-09-16T13:16:36.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:36.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4519 2024-09-16T13:16:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:36.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:37.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4523 2024-09-16T13:16:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:16:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:16:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:16:37.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:37.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:16:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:16:37.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:16:38.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4528 2024-09-16T13:16:38.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4529 2024-09-16T13:16:38.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:16:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:16:38.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:16:39.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4534 2024-09-16T13:16:39.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4533 2024-09-16T13:16:39.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:39.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:16:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:16:39.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:16:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:16:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:16:39.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:16:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:16:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 98104 2024-09-16T13:16:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:39.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:16:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:16:39.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:16:40.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4537 2024-09-16T13:16:40.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:16:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:16:40.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:16:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4539 2024-09-16T13:16:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:16:41.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 98263 2024-09-16T13:16:41.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4540 2024-09-16T13:16:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:16:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:16:41.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:16:41.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:41.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4541 2024-09-16T13:16:41.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:42.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4545 2024-09-16T13:16:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:16:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:16:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:16:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:16:42.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:16:42.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:16:42.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:16:43.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 98417 2024-09-16T13:16:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4547 2024-09-16T13:16:43.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4549 2024-09-16T13:16:44.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:16:44.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 98593 2024-09-16T13:16:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:16:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:16:44.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:16:44.430 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4550 2024-09-16T13:16:44.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:16:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:16:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:16:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:16:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:16:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:16:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:16:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:16:46.425 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:16:46.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4553 2024-09-16T13:16:46.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4554 2024-09-16T13:16:46.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:46.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:47.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4557 2024-09-16T13:16:47.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:16:47.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 98769 2024-09-16T13:16:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:16:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:16:47.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:16:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4559 2024-09-16T13:16:48.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:49.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4561 2024-09-16T13:16:49.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:49.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:16:49.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 98925 2024-09-16T13:16:49.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:49.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:16:49.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:16:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:16:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:16:50.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 99155 2024-09-16T13:16:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:16:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:16:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:16:51.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:16:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:16:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:16:51.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4563 2024-09-16T13:16:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:52.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:16:52.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 99392 2024-09-16T13:16:53.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4565 2024-09-16T13:16:53.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:16:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:16:54.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 99591 2024-09-16T13:16:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T13:16:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:16:56.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 97969 2024-09-16T13:19:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:19:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:19:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:19:10.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 98105 2024-09-16T13:19:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:19:12.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:19:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:19:12.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 98264 2024-09-16T13:19:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:19:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:19:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:19:14.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 98418 2024-09-16T13:19:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:19:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:19:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:19:16.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 98594 2024-09-16T13:19:18.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:19:18.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:19:18.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:19:18.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 98770 2024-09-16T13:19:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:19:20.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:19:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:19:20.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 98926 2024-09-16T13:19:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:19:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:19:22.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:19:22.369 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 99156 2024-09-16T13:19:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:19:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:19:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:19:23.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 99393 2024-09-16T13:19:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:19:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:19:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:19:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 99592 2024-09-16T13:19:27.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:19:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:19:27.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T13:19:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:19:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:19:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:19:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 1 2024-09-16T13:19:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:19:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd0 2024-09-16T13:19:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:19:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary1 2024-09-16T13:19:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 2 2024-09-16T13:19:27.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary1 2024-09-16T13:19:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:19:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary1 2024-09-16T13:19:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd1 2024-09-16T13:19:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary2 2024-09-16T13:19:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary2 2024-09-16T13:19:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:19:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:19:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary2 2024-09-16T13:19:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:19:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 3 2024-09-16T13:19:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd2 2024-09-16T13:19:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary3 2024-09-16T13:19:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary3 2024-09-16T13:19:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:19:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:19:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary3 2024-09-16T13:19:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:19:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 4 2024-09-16T13:19:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:19:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:19:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd3 2024-09-16T13:19:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary4 2024-09-16T13:19:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary4 2024-09-16T13:19:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:19:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary4 2024-09-16T13:19:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:19:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:19:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 6 2024-09-16T13:19:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:19:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:19:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd5 2024-09-16T13:19:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary6 2024-09-16T13:19:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 7 2024-09-16T13:19:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary6 2024-09-16T13:19:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:19:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary6 2024-09-16T13:19:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:19:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd6 2024-09-16T13:19:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary7 2024-09-16T13:19:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary7 2024-09-16T13:19:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:19:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary7 2024-09-16T13:19:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:19:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:19:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:19:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 9 2024-09-16T13:19:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:19:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd8 2024-09-16T13:19:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary9 2024-09-16T13:19:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary9 2024-09-16T13:19:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:19:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 10 2024-09-16T13:19:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary9 2024-09-16T13:19:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:19:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 105528 2024-09-16T13:19:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd9 2024-09-16T13:19:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary10 2024-09-16T13:19:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary10 2024-09-16T13:19:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:19:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 5 2024-09-16T13:19:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary10 2024-09-16T13:19:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd4 2024-09-16T13:19:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary5 2024-09-16T13:19:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary5 2024-09-16T13:19:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:19:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary5 2024-09-16T13:19:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 8 2024-09-16T13:19:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd7 2024-09-16T13:19:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary8 2024-09-16T13:19:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary8 2024-09-16T13:19:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:19:27.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary8 2024-09-16T13:19:28.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd1 2024-09-16T13:19:28.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd6 2024-09-16T13:19:28.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:19:28.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:19:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd7 2024-09-16T13:19:28.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:19:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd5 2024-09-16T13:19:28.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:19:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:19:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd2 2024-09-16T13:19:28.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd8 2024-09-16T13:19:28.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:19:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd3 2024-09-16T13:19:28.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:19:28.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd0 2024-09-16T13:19:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:19:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:19:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd9 2024-09-16T13:19:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:19:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd4 2024-09-16T13:20:15.836 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:20:15.823+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e25b3520 2024-09-16T13:20:15.836 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:20:15.823+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e36aa000 2024-09-16T13:20:15.836 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:20:15.823+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e36ab1e0 2024-09-16T13:20:15.839 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:20:15.827+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e5d9d2b0 2024-09-16T13:20:15.840 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:20:15.827+0000 7f315ffe6700 -1 reset not still connected to 0x55c1f4bf1ba0 2024-09-16T13:20:15.840 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:20:15.827+0000 7f315ffe6700 -1 reset not still connected to 0x55c1f51b4820 2024-09-16T13:20:15.840 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:20:15.827+0000 7f315ffe6700 -1 reset not still connected to 0x55c1fd320dd0 2024-09-16T13:20:15.840 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:20:15.827+0000 7f315ffe6700 -1 reset not still connected to 0x55c1fe127520 2024-09-16T13:20:15.840 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:20:15.827+0000 7f315ffe6700 -1 reset not still connected to 0x55c1fe41c410 2024-09-16T13:20:15.840 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:20:15.831+0000 7f315ffe6700 -1 reset not still connected to 0x55c20d57bad0 2024-09-16T13:20:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=07a3947f768a07d0f19e4ddcc4415173 2024-09-16T13:20:24.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary8 2024-09-16T13:20:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=c837cb416036b67bf3df87aae4aeaba5 2024-09-16T13:20:24.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary7 2024-09-16T13:20:24.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=4836552e52b9530b690e0ffc6dee26fa 2024-09-16T13:20:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary10 2024-09-16T13:20:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=0f2ea2b5a70e3ca03c120b977c3ef0ce 2024-09-16T13:20:25.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary2 2024-09-16T13:20:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=167ea00ad7890e5d1d6cac422940e267 2024-09-16T13:20:25.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary4 2024-09-16T13:20:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=70db66f3e53400a5ca4ea2709a656b37 2024-09-16T13:20:25.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary9 2024-09-16T13:20:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=5d94bf6f3b060a62bf7c0c0689305f3b 2024-09-16T13:20:25.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary1 2024-09-16T13:20:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-16T13:20:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:20:25.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-16T13:20:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-16T13:20:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:20:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-16T13:20:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=d7c45aed17b96c5dccf7580e519673ca 2024-09-16T13:20:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary3 2024-09-16T13:20:26.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-16T13:20:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:20:26.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-16T13:20:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-16T13:20:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:20:26.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-16T13:20:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-16T13:20:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:20:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-16T13:20:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=5958a4b754f0bc5780980368dec09029 2024-09-16T13:20:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary5 2024-09-16T13:20:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=54b2b512625519238b38fcc0709a8993 2024-09-16T13:20:26.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary6 2024-09-16T13:20:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-16T13:20:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:20:26.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-16T13:20:26.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-16T13:20:26.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:26.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:26.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:20:26.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-16T13:20:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-16T13:20:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:26.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:20:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-16T13:20:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-16T13:20:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:20:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-16T13:20:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-16T13:20:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:20:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-16T13:20:27.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:20:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:20:27.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:20:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-16T13:20:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:20:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:27.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:27.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:20:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-16T13:20:27.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:20:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:27.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:20:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-16T13:20:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:20:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:27.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:27.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:20:27.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:20:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-16T13:20:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:20:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:27.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:20:28.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:20:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:20:28.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:20:28.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:20:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-16T13:20:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:20:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:28.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:20:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-16T13:20:28.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:20:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:20:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-16T13:20:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:20:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:20:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:20:28.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:20:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-16T13:20:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:20:28.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-16T13:20:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T13:20:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:28.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:20:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-16T13:20:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:28.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:20:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T13:20:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-16T13:20:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:20:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:28.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:20:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T13:20:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-16T13:20:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:20:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:28.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:20:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-16T13:20:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:20:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:28.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:20:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-16T13:20:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:20:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:28.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T13:20:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:20:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156 2024-09-16T13:20:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:20:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:20:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:23 2024-09-16T13:20:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:20:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:20:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:27' 2024-09-16T13:20:28.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:20:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: bf544298-988b-41f0-a03c-856c55772ccd 2024-09-16T13:20:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:20:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:20:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:23 2024-09-16T13:20:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:20:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:20:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:27' 2024-09-16T13:20:28.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:20:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 3b49bdc8-de30-4087-b019-4db5f1999408 2024-09-16T13:20:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:20:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:20:28.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:23 2024-09-16T13:20:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:20:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:20:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:27' 2024-09-16T13:20:28.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:20:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:20:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:20:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:20:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:20:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:20:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:20:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: e78da270-e064-4749-9b3c-5b1059782eb5 2024-09-16T13:20:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:20:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:20:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:23 2024-09-16T13:20:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:20:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:20:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:27' 2024-09-16T13:20:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:20:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:20:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-16T13:20:29.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:20:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:29.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T13:20:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-16T13:20:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:20:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-16T13:20:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:20:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:29.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T13:20:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:29.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T13:20:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-16T13:20:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:29.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:20:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:29.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T13:20:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-16T13:20:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:20:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:29.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-16T13:20:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:20:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:29.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T13:20:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:29.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T13:20:29.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:20:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 503485e0-28c1-4dd2-9faa-ff8fcbad267f 2024-09-16T13:20:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28 2024-09-16T13:20:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:20:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28' 2024-09-16T13:20:29.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:20:29.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:20:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:20:29.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:20:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:20:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:20:29.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 6e9c1af9-867d-4cc5-b8a8-a0083566b5f4 2024-09-16T13:20:29.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:29.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:29.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:29.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28 2024-09-16T13:20:29.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:29.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:20:29.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:29.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:29.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28' 2024-09-16T13:20:29.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:20:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a5f856a7-332e-4bbf-9058-5deb68a13aee 2024-09-16T13:20:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28 2024-09-16T13:20:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:20:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28' 2024-09-16T13:20:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:20:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:20:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:20:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:20:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:20:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:20:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:20:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:20:29.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:20:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:20:29.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:20:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:29.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:29.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:20:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:20:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:20:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:20:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:20:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:20:29.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:20:29.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:29.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:20:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:20:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:20:29.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:20:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:29.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-16 13:20:28' 2024-09-16T13:20:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-16 13:20:28' 2024-09-16T13:20:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:20:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:20:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:20:28' 2024-09-16T13:20:29.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:20:28' 2024-09-16T13:20:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-16 13:20:28' 2024-09-16T13:20:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:20:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:20:28' 2024-09-16T13:20:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:20:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:29.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:20:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:20:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:20:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 2c59192a-bbce-40d9-8377-cfe3d85c4f4d 2024-09-16T13:20:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28 2024-09-16T13:20:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:20:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:20:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:29' 2024-09-16T13:20:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:20:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 3c6cd560-e2c0-4db8-aea1-e5d13b858250 2024-09-16T13:20:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28 2024-09-16T13:20:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:20:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:20:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:29' 2024-09-16T13:20:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:20:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: d0955fd5-aa21-42b5-8d3d-c814e36c8838 2024-09-16T13:20:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28 2024-09-16T13:20:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:20:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:20:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:29' 2024-09-16T13:20:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:20:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:20:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:20:29.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:20:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:20:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:20:29.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:20:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:20:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:20:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:20:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:20:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:20:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:20:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:20:29.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:20:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:20:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:20:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:20:29.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:20:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:20:29.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:20:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-16 13:20:28' 2024-09-16T13:20:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:20:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:20:28' 2024-09-16T13:20:29.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:20:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:20:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:20:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:20:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-16 13:20:28' 2024-09-16T13:20:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:29.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:20:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:20:28' 2024-09-16T13:20:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:29.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:20:29.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:20:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:20:29.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:20:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-16 13:20:28' 2024-09-16T13:20:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:20:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:20:28' 2024-09-16T13:20:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:29.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:20:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29' 2024-09-16T13:20:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:20:29.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T13:20:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29' 2024-09-16T13:20:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:20:29.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T13:20:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29' 2024-09-16T13:20:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:20:29.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T13:20:29.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29' 2024-09-16T13:20:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:20:29.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T13:20:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T13:20:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T13:20:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T13:20:29.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T13:20:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:20:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:20:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:20:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:20:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:29.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:20:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:20:29.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-16T13:20:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:20:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:20:29.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:20:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:20:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-16T13:20:29.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:20:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:20:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-16T13:20:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:20:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:20:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-16T13:20:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:20:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:20:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29' 2024-09-16T13:20:30.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:20:30.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T13:20:30.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:30.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T13:20:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:20:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:30.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:20:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-16T13:20:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:20:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:30.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:20:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29' 2024-09-16T13:20:30.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:20:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T13:20:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:30.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T13:20:30.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:20:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:30.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:20:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-16T13:20:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:20:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:30.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:20:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-16T13:20:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-16T13:20:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:20:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:20:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-16T13:20:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:20:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T13:20:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T13:20:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T13:20:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-16T13:20:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:20:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T13:20:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:20:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156 2024-09-16T13:20:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:30.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28 2024-09-16T13:20:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:20:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28' 2024-09-16T13:20:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:20:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 3b49bdc8-de30-4087-b019-4db5f1999408 2024-09-16T13:20:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28 2024-09-16T13:20:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:30.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:20:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28' 2024-09-16T13:20:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:20:30.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:20:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:20:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:20:30.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:20:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:20:30.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:20:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:20:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:20:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:20:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:20:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: bf544298-988b-41f0-a03c-856c55772ccd 2024-09-16T13:20:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28 2024-09-16T13:20:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:20:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28' 2024-09-16T13:20:30.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:30.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:20:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:20:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:20:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:20:30.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:20:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:20:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:20:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:20:30.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:20:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:20:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:20:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:20:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:30.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:20:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:30.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:20:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-16 13:20:28' 2024-09-16T13:20:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:20:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:20:28' 2024-09-16T13:20:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:30.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:20:30.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:20:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:20:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-16 13:20:28' 2024-09-16T13:20:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:20:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:20:28' 2024-09-16T13:20:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:20:30.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:20:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-16 13:20:28' 2024-09-16T13:20:30.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:20:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:20:28' 2024-09-16T13:20:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:30.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:20:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:20:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: e78da270-e064-4749-9b3c-5b1059782eb5 2024-09-16T13:20:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28 2024-09-16T13:20:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:20:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28' 2024-09-16T13:20:30.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:30.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:20:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:20:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:20:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:20:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:30.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:20:30.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:30.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:20:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:20:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:20:30.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:20:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-16 13:20:28' 2024-09-16T13:20:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:20:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:30.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:20:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:20:28' 2024-09-16T13:20:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:30.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:20:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-16T13:20:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:30.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:20:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-16T13:20:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:20:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-16T13:20:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-16T13:20:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:20:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-16T13:20:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:20:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:30.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-16T13:20:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-16T13:20:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:30.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-16T13:20:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29' 2024-09-16T13:20:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:20:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T13:20:30.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:30.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T13:20:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:20:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:20:30.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-16T13:20:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:20:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:30.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:30.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:20:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29' 2024-09-16T13:20:30.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:20:30.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T13:20:30.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:30.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T13:20:30.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:20:30.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:30.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:20:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:30.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-16T13:20:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:20:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:30.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:20:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:20:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 2c59192a-bbce-40d9-8377-cfe3d85c4f4d 2024-09-16T13:20:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:20:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:29 2024-09-16T13:20:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:30.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:20:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28' 2024-09-16T13:20:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:20:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 6e9c1af9-867d-4cc5-b8a8-a0083566b5f4 2024-09-16T13:20:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28 2024-09-16T13:20:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:20:30.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28' 2024-09-16T13:20:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:20:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 503485e0-28c1-4dd2-9faa-ff8fcbad267f 2024-09-16T13:20:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28 2024-09-16T13:20:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:20:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:30.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28' 2024-09-16T13:20:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:20:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a5f856a7-332e-4bbf-9058-5deb68a13aee 2024-09-16T13:20:30.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28 2024-09-16T13:20:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:20:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28' 2024-09-16T13:20:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:20:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:20:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:20:30.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:20:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:20:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:20:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:20:30.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:20:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:20:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:20:30.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:20:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:20:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:20:30.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:20:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:20:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:20:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:20:30.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:20:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:20:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:20:30.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:20:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:20:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:20:30.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:20:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:20:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:20:30.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:20:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:20:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:20:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:20:30.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:20:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:20:30.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:20:30.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:20:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:20:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:20:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-16 13:20:28' 2024-09-16T13:20:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:20:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-16 13:20:28' 2024-09-16T13:20:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:20:28' 2024-09-16T13:20:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:20:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:20:28' 2024-09-16T13:20:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:20:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:20:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:29' 2024-09-16T13:20:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-16 13:20:29' 2024-09-16T13:20:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:20:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:20:29' 2024-09-16T13:20:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-16 13:20:28' 2024-09-16T13:20:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:20:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:20:28' 2024-09-16T13:20:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:20:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:20:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-16T13:20:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:20:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:31.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:31.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-16T13:20:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-16T13:20:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:20:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-16T13:20:31.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29' 2024-09-16T13:20:31.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:20:31.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T13:20:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T13:20:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:20:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:20:28cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:20:29' 2024-09-16T13:20:31.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:20:31.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T13:20:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T13:20:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:20:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:20:31.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:20:31.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-16T13:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:31.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:20:31.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:20:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:31.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-16T13:20:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:31.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:20:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:20:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:20:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28' 2024-09-16T13:20:31.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:20:31.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T13:20:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:31.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T13:20:31.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:20:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:31.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:20:31.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-16T13:20:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:20:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:20:31.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-16T13:20:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28' 2024-09-16T13:20:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:20:31.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T13:20:31.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:31.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T13:20:31.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:20:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:20:31.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-16T13:20:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:20:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:20:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-16T13:20:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28' 2024-09-16T13:20:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:20:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T13:20:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28' 2024-09-16T13:20:31.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:20:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:29' 2024-09-16T13:20:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T13:20:31.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T13:20:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:20:31.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T13:20:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:20:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:31.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:20:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:20:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-16T13:20:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:20:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:20:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-16T13:20:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-16T13:20:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:20:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:20:31.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-16T13:20:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:20:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: d0955fd5-aa21-42b5-8d3d-c814e36c8838 2024-09-16T13:20:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:20:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:29 2024-09-16T13:20:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:20:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28' 2024-09-16T13:20:31.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:20:31.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:20:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:20:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:20:31.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 3c6cd560-e2c0-4db8-aea1-e5d13b858250 2024-09-16T13:20:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:20:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:29 2024-09-16T13:20:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:20:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28' 2024-09-16T13:20:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:20:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:31.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:20:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:20:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:20:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:20:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:20:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:20:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:31.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:20:31.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:31.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:20:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:20:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:20:31.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:20:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:29' 2024-09-16T13:20:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-16 13:20:29' 2024-09-16T13:20:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:20:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:20:29' 2024-09-16T13:20:31.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:20:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:20:31.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:20:31.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:20:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:29' 2024-09-16T13:20:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-16 13:20:29' 2024-09-16T13:20:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:20:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:20:29' 2024-09-16T13:20:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:20:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28' 2024-09-16T13:20:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:20:31.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T13:20:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:29' 2024-09-16T13:20:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T13:20:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28' 2024-09-16T13:20:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:20:31.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T13:20:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:20:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:20:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:29' 2024-09-16T13:20:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T13:20:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-16T13:20:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:31.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:20:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:20:31.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-16T13:20:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:20:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:20:31.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:31.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-16T13:20:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:31.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:20:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:20:31.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-16T13:20:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:20:31.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:20:31.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:20:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:20:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:20:31.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:20:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:20:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary5 2024-09-16T13:20:31.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary4 2024-09-16T13:20:31.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:20:31.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary3 2024-09-16T13:20:31.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:20:31.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary1 2024-09-16T13:20:31.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:20:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary6 2024-09-16T13:20:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd0 2024-09-16T13:20:31.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd0 2024-09-16T13:20:31.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:20:31.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd1 2024-09-16T13:20:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:20:31.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd1 2024-09-16T13:20:31.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:20:31.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:20:31.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary9 2024-09-16T13:20:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd2 2024-09-16T13:20:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:20:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd2 2024-09-16T13:20:31.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd3 2024-09-16T13:20:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:20:31.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd3 2024-09-16T13:20:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-16T13:20:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:20:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:31.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-16T13:20:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-16T13:20:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:20:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-16T13:20:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd4 2024-09-16T13:20:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd4 2024-09-16T13:20:31.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:20:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:20:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: bf544298-988b-41f0-a03c-856c55772ccd 2024-09-16T13:20:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28 2024-09-16T13:20:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:20:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:32.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28' 2024-09-16T13:20:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd5 2024-09-16T13:20:32.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:20:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:20:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd5 2024-09-16T13:20:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:20:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:20:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:20:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:20:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:20:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156 2024-09-16T13:20:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28 2024-09-16T13:20:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:20:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28' 2024-09-16T13:20:32.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:32.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:20:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:20:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:20:32.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:20:32.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:20:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:20:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:20:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:20:32.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:32.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:20:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-16 13:20:28' 2024-09-16T13:20:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:20:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:20:28' 2024-09-16T13:20:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:32.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:20:32.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:20:32.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:20:32.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:20:32.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:20:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-16 13:20:28' 2024-09-16T13:20:32.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:20:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:20:28' 2024-09-16T13:20:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:32.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:20:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-16T13:20:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:20:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:32.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:32.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-16T13:20:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-16T13:20:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:20:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:20:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:20:32.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:32.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:32.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-16T13:20:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28' 2024-09-16T13:20:32.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:20:32.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T13:20:32.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28' 2024-09-16T13:20:32.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:20:32.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T13:20:32.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T13:20:32.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T13:20:32.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:20:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:20:32.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:32.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:20:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:32.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-16T13:20:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:20:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:20:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-16T13:20:32.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:20:32.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-16T13:20:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:20:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:20:32.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-16T13:20:32.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:20:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: e78da270-e064-4749-9b3c-5b1059782eb5 2024-09-16T13:20:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28 2024-09-16T13:20:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:20:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28' 2024-09-16T13:20:32.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:20:32.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:20:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:20:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 3b49bdc8-de30-4087-b019-4db5f1999408 2024-09-16T13:20:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:20:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28 2024-09-16T13:20:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:20:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:20:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:20:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:20:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:20:28' 2024-09-16T13:20:32.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:20:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:20:32.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:20:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:20:32.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:20:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:20:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:20:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:32.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:20:32.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:20:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:20:32.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:20:32.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:20:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:32.412 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:20:32.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:20:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:20:32.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:20:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:20:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:32.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-16 13:20:28' 2024-09-16T13:20:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:20:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:20:28' 2024-09-16T13:20:32.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-16 13:20:28' 2024-09-16T13:20:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:32.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:20:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:20:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:20:28' 2024-09-16T13:20:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:20:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:32.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:20:32.426 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:20:32.427 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:20:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:20:32.633 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:20:32.619+0000 7f315ffe6700 -1 reset not still connected to 0x55c20065f930 2024-09-16T13:20:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:20:32.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary10 2024-09-16T13:20:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28' 2024-09-16T13:20:32.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:20:32.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T13:20:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T13:20:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:20:28cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:20:29cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:20:28' 2024-09-16T13:20:32.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:20:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:20:32.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T13:20:32.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:32.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:20:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-16T13:20:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:20:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:20:32.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-16T13:20:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:20:28' 2024-09-16T13:20:32.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T13:20:32.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:20:32.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:20:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:20:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:20:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:20:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-16T13:20:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:20:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:20:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:20:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:20:32.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-16T13:20:33.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd6 2024-09-16T13:20:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:20:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd6 2024-09-16T13:20:33.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:20:33.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:20:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:20:33.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary2 2024-09-16T13:20:33.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:20:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary8 2024-09-16T13:20:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:20:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:20:33.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary7 2024-09-16T13:20:33.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd7 2024-09-16T13:20:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:20:33.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd7 2024-09-16T13:20:33.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd8 2024-09-16T13:20:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:20:34.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd8 2024-09-16T13:20:34.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd9 2024-09-16T13:20:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:20:34.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd9 2024-09-16T13:22:03.431 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:22:03.423+0000 7f74def64700 -1 reset not still connected to 0x559463964c30 2024-09-16T13:22:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=54b2b512625519238b38fcc0709a8993 2024-09-16T13:22:07.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-09-16T13:22:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=167ea00ad7890e5d1d6cac422940e267 2024-09-16T13:22:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd1 2024-09-16T13:22:07.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=70db66f3e53400a5ca4ea2709a656b37 2024-09-16T13:22:07.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd5 2024-09-16T13:22:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=5958a4b754f0bc5780980368dec09029 2024-09-16T13:22:08.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd4 2024-09-16T13:22:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=4836552e52b9530b690e0ffc6dee26fa 2024-09-16T13:22:09.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd6 2024-09-16T13:22:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=d7c45aed17b96c5dccf7580e519673ca 2024-09-16T13:22:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd2 2024-09-16T13:22:09.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 70db66f3e53400a5ca4ea2709a656b37 != \7\0\d\b\6\6\f\3\e\5\3\4\0\0\a\5\c\a\4\e\a\2\7\0\9\a\6\5\6\b\3\7 ]] 2024-09-16T13:22:09.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 54b2b512625519238b38fcc0709a8993 != \5\4\b\2\b\5\1\2\6\2\5\5\1\9\2\3\8\b\3\8\f\c\c\0\7\0\9\a\8\9\9\3 ]] 2024-09-16T13:22:09.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 167ea00ad7890e5d1d6cac422940e267 != \1\6\7\e\a\0\0\a\d\7\8\9\0\e\5\d\1\d\6\c\a\c\4\2\2\9\4\0\e\2\6\7 ]] 2024-09-16T13:22:10.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 5958a4b754f0bc5780980368dec09029 != \5\9\5\8\a\4\b\7\5\4\f\0\b\c\5\7\8\0\9\8\0\3\6\8\d\e\c\0\9\0\2\9 ]] 2024-09-16T13:22:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=5d94bf6f3b060a62bf7c0c0689305f3b 2024-09-16T13:22:10.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd3 2024-09-16T13:22:10.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=07a3947f768a07d0f19e4ddcc4415173 2024-09-16T13:22:10.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd8 2024-09-16T13:22:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=0f2ea2b5a70e3ca03c120b977c3ef0ce 2024-09-16T13:22:10.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd7 2024-09-16T13:22:10.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 4836552e52b9530b690e0ffc6dee26fa != \4\8\3\6\5\5\2\e\5\2\b\9\5\3\0\b\6\9\0\e\0\f\f\c\6\d\e\e\2\6\f\a ]] 2024-09-16T13:22:10.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ d7c45aed17b96c5dccf7580e519673ca != \d\7\c\4\5\a\e\d\1\7\b\9\6\c\5\d\c\c\f\7\5\8\0\e\5\1\9\6\7\3\c\a ]] 2024-09-16T13:22:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=c837cb416036b67bf3df87aae4aeaba5 2024-09-16T13:22:10.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd9 2024-09-16T13:22:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 5d94bf6f3b060a62bf7c0c0689305f3b != \5\d\9\4\b\f\6\f\3\b\0\6\0\a\6\2\b\f\7\c\0\c\0\6\8\9\3\0\5\f\3\b ]] 2024-09-16T13:22:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:22:10.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 105529 2024-09-16T13:22:10.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 07a3947f768a07d0f19e4ddcc4415173 != \0\7\a\3\9\4\7\f\7\6\8\a\0\7\d\0\f\1\9\e\4\d\d\c\c\4\4\1\5\1\7\3 ]] 2024-09-16T13:22:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 0f2ea2b5a70e3ca03c120b977c3ef0ce != \0\f\2\e\a\2\b\5\a\7\0\e\3\c\a\0\3\c\1\2\0\b\9\7\7\c\3\e\f\0\c\e ]] 2024-09-16T13:22:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:22:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 105530 2024-09-16T13:22:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:22:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 105533 2024-09-16T13:22:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:22:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 105535 2024-09-16T13:22:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:22:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 105536 2024-09-16T13:22:10.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:22:10.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 105538 2024-09-16T13:22:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ c837cb416036b67bf3df87aae4aeaba5 != \c\8\3\7\c\b\4\1\6\0\3\6\b\6\7\b\f\3\d\f\8\7\a\a\e\4\a\e\a\b\a\5 ]] 2024-09-16T13:22:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:22:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 105539 2024-09-16T13:22:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:22:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 105542 2024-09-16T13:22:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:22:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 105543 2024-09-16T13:22:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T13:22:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:22:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary1 2024-09-16T13:22:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-16T13:22:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:22:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:22:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:22:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:22:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:22:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-16T13:22:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:12.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:22:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:22:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:22:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:22:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:22:12.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T13:22:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:22:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 2c59192a-bbce-40d9-8377-cfe3d85c4f4d 2024-09-16T13:22:12.367 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:22:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":149860352.0,"last_snapshot_bytes":149860352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"} 2024-09-16T13:22:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:22:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:21:53 2024-09-16T13:22:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:22:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:22:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:22:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:22:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:22:08' 2024-09-16T13:22:12.368 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:22:12.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:22:12.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:22:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":149860352.0,"last_snapshot_bytes":149860352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"} 2024-09-16T13:22:12.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:22:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:22:12.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:22:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:22:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:22:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:22:12.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:22:12.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:22:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:21:53' 2024-09-16T13:22:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-16 13:21:53' 2024-09-16T13:22:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:22:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:21:53' 2024-09-16T13:22:12.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:22:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:22:12.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:22:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary12c59192a-bbce-40d9-8377-cfe3d85c4f4dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":149860352.0,"last_snapshot_bytes":149860352,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:08image-primary10bf544298-988b-41f0-a03c-856c55772ccdup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492832,"remote_snapshot_timestamp":1726492832,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:08image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62078976.0,"last_snapshot_bytes":62078976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:08image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65191936.0,"last_snapshot_bytes":65191936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:08image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72261632.0,"last_snapshot_bytes":72261632,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:08image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70148096.0,"last_snapshot_bytes":70148096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:08image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62173184.0,"last_snapshot_bytes":62173184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:08image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65257472.0,"last_snapshot_bytes":65257472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:08image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62226432.0,"last_snapshot_bytes":62226432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:08image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60592128.0,"last_snapshot_bytes":60592128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:08' 2024-09-16T13:22:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:22:13.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T13:22:13.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:21:53' 2024-09-16T13:22:13.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T13:22:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:22:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:22:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:22:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:22:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-16T13:22:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:22:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:22:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-16T13:22:13.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-16T13:22:15.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:22:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-16T13:22:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:22:15.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:22:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:22:15.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary1 2024-09-16T13:22:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary1 2024-09-16T13:22:15.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:22:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary2 2024-09-16T13:22:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-16T13:22:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:22:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:22:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:22:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:22:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:22:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-16T13:22:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:16.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:22:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:22:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:22:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:22:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:22:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T13:22:16.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:22:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156 2024-09-16T13:22:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:22:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62078976.0,"last_snapshot_bytes":62078976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"} 2024-09-16T13:22:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:22:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:21:53 2024-09-16T13:22:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:22:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:22:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:22:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:22:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:22:13' 2024-09-16T13:22:16.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:22:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:22:16.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:22:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62078976.0,"last_snapshot_bytes":62078976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"} 2024-09-16T13:22:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:22:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:22:16.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:22:16.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:22:16.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:22:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:22:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:22:16.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:22:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:21:53' 2024-09-16T13:22:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-16 13:21:53' 2024-09-16T13:22:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:22:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:21:53' 2024-09-16T13:22:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:22:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:22:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:22:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK942544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10bf544298-988b-41f0-a03c-856c55772ccdup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492832,"remote_snapshot_timestamp":1726492832,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:13image-primary2f5fb41d1-7c92-4e7c-aaa4-1fcf78b2e156up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62078976.0,"last_snapshot_bytes":62078976,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:13image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65191936.0,"last_snapshot_bytes":65191936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:13image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72261632.0,"last_snapshot_bytes":72261632,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:13image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70148096.0,"last_snapshot_bytes":70148096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:13image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62173184.0,"last_snapshot_bytes":62173184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:13image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65257472.0,"last_snapshot_bytes":65257472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:13image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62226432.0,"last_snapshot_bytes":62226432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:13image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60592128.0,"last_snapshot_bytes":60592128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:13' 2024-09-16T13:22:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:22:17.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T13:22:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:21:53' 2024-09-16T13:22:17.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T13:22:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:22:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:22:17.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:22:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:22:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-16T13:22:17.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:22:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:22:17.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-16T13:22:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-16T13:22:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:22:18.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-16T13:22:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:22:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:22:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:22:18.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary2 2024-09-16T13:22:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary2 2024-09-16T13:22:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:22:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary3 2024-09-16T13:22:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-16T13:22:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:22:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:22:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:22:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:22:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:22:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-16T13:22:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:19.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:22:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:22:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:22:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:22:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:22:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T13:22:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:22:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 503485e0-28c1-4dd2-9faa-ff8fcbad267f 2024-09-16T13:22:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:22:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65191936.0,"last_snapshot_bytes":65191936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"} 2024-09-16T13:22:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:22:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:21:53 2024-09-16T13:22:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:22:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:22:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:22:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:22:19.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:22:18' 2024-09-16T13:22:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:22:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:22:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:22:19.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65191936.0,"last_snapshot_bytes":65191936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"} 2024-09-16T13:22:19.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:22:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:22:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:22:19.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:22:19.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:22:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:22:19.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:22:19.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:22:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:21:53' 2024-09-16T13:22:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-16 13:21:53' 2024-09-16T13:22:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:19.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:22:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:21:53' 2024-09-16T13:22:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:22:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:22:19.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:22:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK842544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10bf544298-988b-41f0-a03c-856c55772ccdup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492832,"remote_snapshot_timestamp":1726492832,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:18image-primary3503485e0-28c1-4dd2-9faa-ff8fcbad267fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65191936.0,"last_snapshot_bytes":65191936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:18image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72261632.0,"last_snapshot_bytes":72261632,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:18image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70148096.0,"last_snapshot_bytes":70148096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:18image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62173184.0,"last_snapshot_bytes":62173184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:18image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65257472.0,"last_snapshot_bytes":65257472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:18image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62226432.0,"last_snapshot_bytes":62226432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:18image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60592128.0,"last_snapshot_bytes":60592128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:18' 2024-09-16T13:22:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:22:19.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T13:22:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:21:53' 2024-09-16T13:22:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T13:22:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:22:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:22:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:22:19.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:22:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-16T13:22:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:22:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:22:19.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-16T13:22:20.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-16T13:22:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:22:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-16T13:22:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:22:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:22:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:22:21.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary3 2024-09-16T13:22:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:21.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary3 2024-09-16T13:22:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:22:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary4 2024-09-16T13:22:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-16T13:22:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:22:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:22:21.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:22:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:22:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:22:21.821 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:22:21.814+0000 7ff4f6946700 -1 reset not still connected to 0x55755b91f790 2024-09-16T13:22:21.822 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:22:21.814+0000 7ff4f6946700 -1 reset not still connected to 0x55755b91fd40 2024-09-16T13:22:21.896 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:22:21.886+0000 7f61b1855700 -1 reset not still connected to 0x557b229664e0 2024-09-16T13:22:21.896 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:22:21.886+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T13:22:21.896 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:22:21.886+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T13:22:21.896 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:22:21.886+0000 7f61b1855700 -1 reset not still connected to 0x557b3bf8dc70 2024-09-16T13:22:21.896 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:22:21.886+0000 7f61b1855700 -1 reset not still connected to 0x557b3f9aed00 2024-09-16T13:22:21.896 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:22:21.886+0000 7f61b1855700 -1 reset not still connected to 0x557b3f9af110 2024-09-16T13:22:21.896 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:22:21.886+0000 7f61b1855700 -1 reset not still connected to 0x557b3fd72820 2024-09-16T13:22:22.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-16T13:22:22.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:22.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:22.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:22:22.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:22.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:22:22.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:22:22.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:22:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:22:22.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T13:22:22.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:22:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 6e9c1af9-867d-4cc5-b8a8-a0083566b5f4 2024-09-16T13:22:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:22:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72261632.0,"last_snapshot_bytes":72261632,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"} 2024-09-16T13:22:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:22:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:21:53 2024-09-16T13:22:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:22:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:22:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:22:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:22:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:22:18' 2024-09-16T13:22:22.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:22:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:22:22.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:22:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72261632.0,"last_snapshot_bytes":72261632,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"} 2024-09-16T13:22:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:22:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:22:22.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:22:22.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:22:22.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:22:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:22:22.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:22:22.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:22:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:21:53' 2024-09-16T13:22:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-16 13:21:53' 2024-09-16T13:22:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:22:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:21:53' 2024-09-16T13:22:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:22:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:22:22.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:22:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK742544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10bf544298-988b-41f0-a03c-856c55772ccdup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492832,"remote_snapshot_timestamp":1726492832,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:18image-primary46e9c1af9-867d-4cc5-b8a8-a0083566b5f4up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72261632.0,"last_snapshot_bytes":72261632,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:18image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70148096.0,"last_snapshot_bytes":70148096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:18image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62173184.0,"last_snapshot_bytes":62173184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:18image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65257472.0,"last_snapshot_bytes":65257472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:18image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62226432.0,"last_snapshot_bytes":62226432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:18image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60592128.0,"last_snapshot_bytes":60592128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:21:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:18' 2024-09-16T13:22:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:22:22.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T13:22:22.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:21:53' 2024-09-16T13:22:22.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T13:22:22.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:22:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:22:22.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:22:22.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:22:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-16T13:22:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:22:22.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:22:22.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-16T13:22:22.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-16T13:22:24.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:22:24.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-16T13:22:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:22:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:22:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:22:24.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary4 2024-09-16T13:22:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:24.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary4 2024-09-16T13:22:24.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:22:24.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary5 2024-09-16T13:22:24.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-16T13:22:24.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:24.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:24.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:22:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:22:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:22:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:22:24.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:22:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-16T13:22:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:22:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:22:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:22:25.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:22:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:22:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T13:22:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:22:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a5f856a7-332e-4bbf-9058-5deb68a13aee 2024-09-16T13:22:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:22:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70148096.0,"last_snapshot_bytes":70148096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"} 2024-09-16T13:22:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:22:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:22:23 2024-09-16T13:22:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:22:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:22:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:22:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:22:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:22:23' 2024-09-16T13:22:25.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:22:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:22:25.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:22:25.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70148096.0,"last_snapshot_bytes":70148096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"} 2024-09-16T13:22:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:22:25.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:22:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:22:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:22:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:22:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:22:25.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:22:25.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:22:25.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:22:23' 2024-09-16T13:22:25.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-16 13:22:23' 2024-09-16T13:22:25.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:25.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:25.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:22:25.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:25.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:22:23' 2024-09-16T13:22:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:22:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:22:25.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:22:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK642544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10bf544298-988b-41f0-a03c-856c55772ccdup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492832,"remote_snapshot_timestamp":1726492832,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:23image-primary5a5f856a7-332e-4bbf-9058-5deb68a13aeeup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70148096.0,"last_snapshot_bytes":70148096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:23image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62173184.0,"last_snapshot_bytes":62173184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:23image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65257472.0,"last_snapshot_bytes":65257472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:23image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62226432.0,"last_snapshot_bytes":62226432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:23image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60592128.0,"last_snapshot_bytes":60592128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:23' 2024-09-16T13:22:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:22:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T13:22:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:22:23' 2024-09-16T13:22:25.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T13:22:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:22:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:22:25.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:22:25.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:22:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:25.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-16T13:22:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:22:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:22:25.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-16T13:22:26.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-16T13:22:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:22:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-16T13:22:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:22:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:22:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:22:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary5 2024-09-16T13:22:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary5 2024-09-16T13:22:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:22:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary6 2024-09-16T13:22:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-16T13:22:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:22:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:22:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:22:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:22:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:22:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-16T13:22:28.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:22:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:22:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:22:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:22:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:22:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T13:22:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:22:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 3c6cd560-e2c0-4db8-aea1-e5d13b858250 2024-09-16T13:22:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:22:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62173184.0,"last_snapshot_bytes":62173184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"} 2024-09-16T13:22:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:22:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:22:23 2024-09-16T13:22:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:22:28.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:22:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:22:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:22:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:22:28' 2024-09-16T13:22:28.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:22:28.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:22:28.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:22:28.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62173184.0,"last_snapshot_bytes":62173184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"} 2024-09-16T13:22:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:22:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:22:28.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:22:28.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:22:28.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:22:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:22:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:22:28.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:22:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:22:23' 2024-09-16T13:22:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-16 13:22:23' 2024-09-16T13:22:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:22:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:22:23' 2024-09-16T13:22:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:22:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:22:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:22:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK542544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10bf544298-988b-41f0-a03c-856c55772ccdup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492832,"remote_snapshot_timestamp":1726492832,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:28image-primary63c6cd560-e2c0-4db8-aea1-e5d13b858250up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62173184.0,"last_snapshot_bytes":62173184,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:28image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65257472.0,"last_snapshot_bytes":65257472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62226432.0,"last_snapshot_bytes":62226432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60592128.0,"last_snapshot_bytes":60592128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:28' 2024-09-16T13:22:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:22:29.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T13:22:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:22:23' 2024-09-16T13:22:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T13:22:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:22:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:22:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:22:29.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:22:29.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-16T13:22:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:22:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:22:29.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-16T13:22:29.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-16T13:22:30.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:22:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-16T13:22:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:22:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:22:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:22:30.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary6 2024-09-16T13:22:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary6 2024-09-16T13:22:30.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:22:30.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary7 2024-09-16T13:22:30.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-16T13:22:30.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:30.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:30.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:22:30.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:30.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:22:30.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:22:30.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:22:30.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:22:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-16T13:22:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:22:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:22:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:22:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:22:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:22:31.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T13:22:31.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:22:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: e78da270-e064-4749-9b3c-5b1059782eb5 2024-09-16T13:22:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:22:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65257472.0,"last_snapshot_bytes":65257472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"} 2024-09-16T13:22:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:22:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:22:23 2024-09-16T13:22:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:22:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:22:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:22:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:22:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:22:28' 2024-09-16T13:22:31.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:22:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:22:31.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:22:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65257472.0,"last_snapshot_bytes":65257472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"} 2024-09-16T13:22:31.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:22:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:22:31.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:22:31.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:22:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:22:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:22:31.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:22:31.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:22:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:22:23' 2024-09-16T13:22:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-16 13:22:23' 2024-09-16T13:22:31.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:22:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:22:23' 2024-09-16T13:22:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:22:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:22:31.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:22:32.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK442544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10bf544298-988b-41f0-a03c-856c55772ccdup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492832,"remote_snapshot_timestamp":1726492832,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:28image-primary7e78da270-e064-4749-9b3c-5b1059782eb5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65257472.0,"last_snapshot_bytes":65257472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:28image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62226432.0,"last_snapshot_bytes":62226432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:28image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60592128.0,"last_snapshot_bytes":60592128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:28' 2024-09-16T13:22:32.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:22:32.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T13:22:32.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:22:23' 2024-09-16T13:22:32.057 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T13:22:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:22:32.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:22:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:22:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:22:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-16T13:22:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:22:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:22:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-16T13:22:32.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-16T13:22:33.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:22:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-16T13:22:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:22:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:22:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:22:33.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary7 2024-09-16T13:22:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary7 2024-09-16T13:22:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:22:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary8 2024-09-16T13:22:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-16T13:22:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:22:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:22:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:22:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:22:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:22:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-16T13:22:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:22:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:34.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:22:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:22:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:22:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:22:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T13:22:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:22:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 3b49bdc8-de30-4087-b019-4db5f1999408 2024-09-16T13:22:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:22:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62226432.0,"last_snapshot_bytes":62226432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"} 2024-09-16T13:22:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:22:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:22:23 2024-09-16T13:22:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:22:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:22:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:22:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:22:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:22:33' 2024-09-16T13:22:34.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:22:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:22:34.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:22:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62226432.0,"last_snapshot_bytes":62226432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"} 2024-09-16T13:22:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:22:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:22:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:22:34.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:22:34.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:22:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:22:34.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:22:34.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:22:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:22:23' 2024-09-16T13:22:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-16 13:22:23' 2024-09-16T13:22:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:22:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:22:23' 2024-09-16T13:22:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:22:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:22:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:22:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK342544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10bf544298-988b-41f0-a03c-856c55772ccdup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492832,"remote_snapshot_timestamp":1726492832,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:33image-primary83b49bdc8-de30-4087-b019-4db5f1999408up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62226432.0,"last_snapshot_bytes":62226432,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492833,"remote_snapshot_timestamp":1726492833,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:33image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60592128.0,"last_snapshot_bytes":60592128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:33' 2024-09-16T13:22:35.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:22:35.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T13:22:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:22:23' 2024-09-16T13:22:35.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T13:22:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:22:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:22:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:22:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:22:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:35.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-16T13:22:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:22:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:22:35.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-16T13:22:35.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-16T13:22:37.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:22:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-16T13:22:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:22:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:22:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:22:37.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary8 2024-09-16T13:22:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:37.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary8 2024-09-16T13:22:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:22:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary9 2024-09-16T13:22:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-16T13:22:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:22:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:22:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:22:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:22:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:22:37.951 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:22:37.941+0000 7ff4f6946700 -1 reset not still connected to 0x55753c1b1380 2024-09-16T13:22:37.951 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:22:37.941+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1748f0 2024-09-16T13:22:37.953 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:22:37.945+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174d00 2024-09-16T13:22:37.955 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:22:37.945+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T13:22:37.958 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:22:37.949+0000 7ff4f6946700 -1 reset not still connected to 0x55755904e4e0 2024-09-16T13:22:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-16T13:22:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:38.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:22:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:22:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:22:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:22:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:22:38.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T13:22:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:22:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: d0955fd5-aa21-42b5-8d3d-c814e36c8838 2024-09-16T13:22:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:22:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60592128.0,"last_snapshot_bytes":60592128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"} 2024-09-16T13:22:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:22:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:22:23 2024-09-16T13:22:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:22:38.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:22:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:22:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:22:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:22:33' 2024-09-16T13:22:38.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:22:38.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:22:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:22:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60592128.0,"last_snapshot_bytes":60592128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"} 2024-09-16T13:22:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:22:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:22:38.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:22:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:22:38.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:22:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:22:38.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:22:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:22:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:22:23' 2024-09-16T13:22:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-16 13:22:23' 2024-09-16T13:22:38.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:22:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:22:23' 2024-09-16T13:22:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:22:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:22:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:22:38.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK242544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10bf544298-988b-41f0-a03c-856c55772ccdup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492832,"remote_snapshot_timestamp":1726492832,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:38image-primary9d0955fd5-aa21-42b5-8d3d-c814e36c8838up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60592128.0,"last_snapshot_bytes":60592128,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492831,"remote_snapshot_timestamp":1726492831,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:38' 2024-09-16T13:22:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:22:38.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T13:22:38.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:22:23' 2024-09-16T13:22:38.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T13:22:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:22:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:22:38.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:22:38.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:22:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-16T13:22:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:22:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:38.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:22:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-16T13:22:38.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-16T13:22:40.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:22:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-16T13:22:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:22:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:22:40.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:22:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary9 2024-09-16T13:22:40.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:40.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary9 2024-09-16T13:22:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:22:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary10 2024-09-16T13:22:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-16T13:22:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:22:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:22:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:22:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:22:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:22:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-16T13:22:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:22:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:22:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:22:41.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:22:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:22:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T13:22:41.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:22:41.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: bf544298-988b-41f0-a03c-856c55772ccd 2024-09-16T13:22:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:22:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492832,"remote_snapshot_timestamp":1726492832,"replay_state":"idle"} 2024-09-16T13:22:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:22:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:22:23 2024-09-16T13:22:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:22:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:22:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:22:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:22:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:22:38' 2024-09-16T13:22:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:22:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:22:41.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:22:41.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492832,"remote_snapshot_timestamp":1726492832,"replay_state":"idle"} 2024-09-16T13:22:41.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:22:41.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:22:41.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:22:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:22:41.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:22:41.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:22:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:22:41.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:22:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:22:23' 2024-09-16T13:22:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-16 13:22:23' 2024-09-16T13:22:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:22:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:22:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:22:23' 2024-09-16T13:22:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:22:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:22:41.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:22:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK142544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10bf544298-988b-41f0-a03c-856c55772ccdup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70119424.0,"last_snapshot_bytes":70119424,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726492832,"remote_snapshot_timestamp":1726492832,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:22:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:22:38' 2024-09-16T13:22:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:22:41.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T13:22:41.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:22:23' 2024-09-16T13:22:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T13:22:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:22:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:22:41.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:22:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:22:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-16T13:22:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:22:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:22:41.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-16T13:22:41.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-16T13:22:43.379 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T13:22:43.369+0000 7f4071c2f700 -1 reset not still connected to 0x563591d7d040 2024-09-16T13:22:43.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:22:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-16T13:22:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:22:43.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:22:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:22:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary10 2024-09-16T13:22:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:22:43.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary10 2024-09-16T13:22:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..10} 2024-09-16T13:22:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS=() 2024-09-16T13:22:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS=() 2024-09-16T13:22:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS=() 2024-09-16T13:22:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS=() 2024-09-16T13:22:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ RET=0 2024-09-16T13:22:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:22:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary1 2024-09-16T13:22:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary1 2024-09-16T13:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-16T13:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-16T13:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:43.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:22:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:22:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:22:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:22:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:43.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-16T13:22:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:22:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-16T13:22:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:22:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:22:43.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-16T13:22:44.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:22:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-16T13:22:44.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary1': 2024-09-16T13:22:44.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:22:44.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:22:44.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:22:44.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 31193a5f8841 2024-09-16T13:22:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.31193a5f8841 2024-09-16T13:22:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:22:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:22:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:22:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:22:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:22:43 2024 2024-09-16T13:22:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:22:43 2024 2024-09-16T13:22:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:22:43 2024 2024-09-16T13:22:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:22:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:22:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: ab41f43b-2d91-4d86-a967-20ff72553573 2024-09-16T13:22:44.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:22:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:22:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary1 2024-09-16T13:22:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd0 2024-09-16T13:22:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:22:44.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd0 2024-09-16T13:22:44.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:22:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/26214402101248/2621440 done 2024-09-16T13:22:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:22:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: d333bfae-8fd1-4441-9950-1f6360856899 2024-09-16T13:22:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:22:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:22:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:22:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:22:45.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:22:45.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:22:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:22:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:22:45.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary1 2024-09-16T13:22:45.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd0 test-primary1 2024-09-16T13:22:45.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:22:45.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary1 2024-09-16T13:22:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:22:45.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-16T13:22:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:22:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:22:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:22:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:22:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:22:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:22:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary2 2024-09-16T13:22:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary2 2024-09-16T13:22:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:22:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-16T13:22:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:22:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:22:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:22:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-16T13:22:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:22:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:22:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:22:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:22:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-16T13:22:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary1 2024-09-16T13:22:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary1 2024-09-16T13:22:45.456 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-16T13:22:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:22:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-16T13:22:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:22:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:22:45.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-16T13:22:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:22:45.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-16T13:22:46.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:22:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4618 2024-09-16T13:22:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:22:46.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-16T13:22:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary2': 2024-09-16T13:22:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:22:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:22:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:22:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 3129a6cd1df 2024-09-16T13:22:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.3129a6cd1df 2024-09-16T13:22:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:22:46.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:22:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:22:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:22:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:22:45 2024 2024-09-16T13:22:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:22:45 2024 2024-09-16T13:22:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:22:45 2024 2024-09-16T13:22:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:22:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:22:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 8dbcffb5-dab3-446c-89d2-a6bb77d01c73 2024-09-16T13:22:46.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:22:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:22:46.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary2 2024-09-16T13:22:46.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd1 2024-09-16T13:22:46.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:22:46.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd1 2024-09-16T13:22:46.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:22:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:22:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:22:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 919047fd-ed6b-418a-89a4-5622075af983 2024-09-16T13:22:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:22:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:22:46.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:22:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:22:46.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:22:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:22:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:22:47.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:22:47.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary2 2024-09-16T13:22:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd1 test-primary2 2024-09-16T13:22:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:22:47.310 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary2 2024-09-16T13:22:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:22:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-16T13:22:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:22:47.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:22:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:22:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary2 2024-09-16T13:22:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary3 2024-09-16T13:22:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary3 2024-09-16T13:22:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:22:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-16T13:22:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:22:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:22:47.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary2 2024-09-16T13:22:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:22:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-16T13:22:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:22:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:22:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:22:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-16T13:22:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:22:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:22:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:22:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:22:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:47.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-16T13:22:47.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:22:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-16T13:22:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:22:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:22:47.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-16T13:22:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:22:47.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-16T13:22:47.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4621 2024-09-16T13:22:47.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:22:47.657 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:22:47.645+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T13:22:47.657 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:22:47.645+0000 7f61b1855700 -1 reset not still connected to 0x557b377d9930 2024-09-16T13:22:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:22:48.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-16T13:22:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary3': 2024-09-16T13:22:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:22:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:22:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:22:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 31398e2b92b7 2024-09-16T13:22:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.31398e2b92b7 2024-09-16T13:22:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:22:48.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:22:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:22:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:22:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:22:47 2024 2024-09-16T13:22:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:22:47 2024 2024-09-16T13:22:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:22:47 2024 2024-09-16T13:22:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:22:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:22:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: da5b1f11-8a62-48f0-8749-2008dd7a75cc 2024-09-16T13:22:48.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:22:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:22:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary3 2024-09-16T13:22:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd2 2024-09-16T13:22:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:22:48.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd2 2024-09-16T13:22:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:22:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:22:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:22:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 48fc6b06-5ad6-40c7-941c-c48af994d4bc 2024-09-16T13:22:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:22:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:22:48.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:22:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:22:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:22:49.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:22:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:22:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:22:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary3 2024-09-16T13:22:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd2 test-primary3 2024-09-16T13:22:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:22:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:22:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:22:49.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:22:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:22:49.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary3 2024-09-16T13:22:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:22:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-16T13:22:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:22:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:49.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:22:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:22:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary3 2024-09-16T13:22:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:22:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary4 2024-09-16T13:22:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary4 2024-09-16T13:22:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:22:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-16T13:22:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:22:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:22:49.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary3 2024-09-16T13:22:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-16T13:22:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:22:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:22:49.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:22:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-16T13:22:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:22:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:22:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:22:49.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:22:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:49.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-16T13:22:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:22:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-16T13:22:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:22:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:22:49.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-16T13:22:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:22:49.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-16T13:22:49.890 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:22:49.881+0000 7f61b1855700 -1 reset not still connected to 0x557b1f47f930 2024-09-16T13:22:49.890 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:22:49.881+0000 7f61b1855700 -1 reset not still connected to 0x557b1f47fd40 2024-09-16T13:22:49.891 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:22:49.885+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T13:22:49.891 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:22:49.885+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T13:22:49.892 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:22:49.885+0000 7f61b1855700 -1 reset not still connected to 0x557b377d9930 2024-09-16T13:22:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:22:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:22:50.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:22:50.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:22:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4623 2024-09-16T13:22:50.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:22:50.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:22:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-16T13:22:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary4': 2024-09-16T13:22:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:22:50.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:22:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:22:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 3149eb770d1e 2024-09-16T13:22:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.3149eb770d1e 2024-09-16T13:22:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:22:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:22:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:22:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:22:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:22:49 2024 2024-09-16T13:22:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:22:49 2024 2024-09-16T13:22:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:22:49 2024 2024-09-16T13:22:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:22:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:22:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: a2ea9fe0-ce1d-400a-8b01-1f096f691b5c 2024-09-16T13:22:50.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:22:50.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:22:50.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary4 2024-09-16T13:22:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd3 2024-09-16T13:22:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:22:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd3 2024-09-16T13:22:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:22:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:22:50.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:22:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: c1b1b647-82f8-4b3b-b089-fd308aac7749 2024-09-16T13:22:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:22:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:22:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:22:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:22:50.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:22:51.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:22:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:22:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:22:51.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary4 2024-09-16T13:22:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd3 test-primary4 2024-09-16T13:22:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4625 2024-09-16T13:22:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:22:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:22:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary4 2024-09-16T13:22:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:22:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-16T13:22:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:22:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:22:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:22:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:22:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:22:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:22:51.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary4 2024-09-16T13:22:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:22:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary5 2024-09-16T13:22:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary5 2024-09-16T13:22:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-16T13:22:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary4 2024-09-16T13:22:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:22:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-16T13:22:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:22:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:22:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-16T13:22:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:22:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:22:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:22:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:22:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-16T13:22:51.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:22:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-16T13:22:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:22:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:22:51.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-16T13:22:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:22:51.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-16T13:22:52.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4627 2024-09-16T13:22:52.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:22:52.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4631 2024-09-16T13:22:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:22:52.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:22:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-16T13:22:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary5': 2024-09-16T13:22:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:22:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:22:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:22:52.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 315ceee7076e 2024-09-16T13:22:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.315ceee7076e 2024-09-16T13:22:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:22:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:22:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:22:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:22:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:22:51 2024 2024-09-16T13:22:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:22:51 2024 2024-09-16T13:22:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:22:51 2024 2024-09-16T13:22:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:22:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:22:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: b48e7af7-8636-448d-b1d6-46b0539ab6e9 2024-09-16T13:22:52.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:22:52.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:22:52.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary5 2024-09-16T13:22:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd4 2024-09-16T13:22:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:22:52.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd4 2024-09-16T13:22:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:22:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:22:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:22:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 5e49c5d8-a634-4719-8719-fe59822c7fb7 2024-09-16T13:22:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:22:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:22:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:22:52.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:22:52.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:22:53.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:22:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:22:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:22:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary5 2024-09-16T13:22:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd4 test-primary5 2024-09-16T13:22:53.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:22:53.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary5 2024-09-16T13:22:53.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:22:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:22:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:22:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary6 2024-09-16T13:22:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary6 2024-09-16T13:22:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-16T13:22:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:22:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:22:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:22:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-16T13:22:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:22:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:22:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary5 2024-09-16T13:22:53.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:22:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:22:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-16T13:22:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary5 2024-09-16T13:22:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-16T13:22:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-16T13:22:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:22:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:22:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:22:53.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:22:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:22:53.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:22:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-16T13:22:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:22:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:22:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-16T13:22:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:22:53.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-16T13:22:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:22:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:22:53.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:22:53.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:22:53.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4634 2024-09-16T13:22:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:22:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:22:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:22:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:22:54.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:22:54.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:22:54.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-16T13:22:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary6': 2024-09-16T13:22:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:22:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:22:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 316b38c5063b 2024-09-16T13:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.316b38c5063b 2024-09-16T13:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:22:53 2024 2024-09-16T13:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:22:53 2024 2024-09-16T13:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:22:53 2024 2024-09-16T13:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 64ea0105-23d9-444f-a0cb-021f5706b8b1 2024-09-16T13:22:54.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:22:54.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:22:54.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary6 2024-09-16T13:22:54.817 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T13:22:54.809+0000 7f4071c2f700 -1 reset not still connected to 0x563582d78f70 2024-09-16T13:22:54.817 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T13:22:54.809+0000 7f4071c2f700 -1 reset not still connected to 0x563591bfbba0 2024-09-16T13:22:54.817 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T13:22:54.809+0000 7f4071c2f700 -1 reset not still connected to 0x563591d7d040 2024-09-16T13:22:54.817 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T13:22:54.809+0000 7f4071c2f700 -1 reset not still connected to 0x56359c9ac5b0 2024-09-16T13:22:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd5 2024-09-16T13:22:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:22:54.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd5 2024-09-16T13:22:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:22:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 528384/2621440 done 2024-09-16T13:22:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:22:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 6fe13db1-aea7-48b3-ab70-0de02371c7c2 2024-09-16T13:22:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:22:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:22:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:22:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:22:55.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:22:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:22:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:22:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:22:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:22:55.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:22:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 Snapshot ID: 4636 2024-09-16T13:22:55.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:22:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:22:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:22:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:22:55.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:22:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T13:22:55.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:22:55.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary6 2024-09-16T13:22:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd5 test-primary6 2024-09-16T13:22:55.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:22:55.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary6 2024-09-16T13:22:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:22:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-16T13:22:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:22:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:22:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:22:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:22:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:22:55.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:22:55.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary6 2024-09-16T13:22:55.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:22:55.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary6 2024-09-16T13:22:55.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary7 2024-09-16T13:22:55.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-16T13:22:55.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary7 2024-09-16T13:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-16T13:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-16T13:22:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:55.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-16T13:22:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:22:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-16T13:22:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:22:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:22:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-16T13:22:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:22:55.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-16T13:22:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4637 2024-09-16T13:22:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:22:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:22:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:22:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:22:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:22:56.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:22:56.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4642 2024-09-16T13:22:56.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:22:56.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-16T13:22:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary7': 2024-09-16T13:22:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:22:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:22:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:22:56.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 3184bae24ad3 2024-09-16T13:22:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.3184bae24ad3 2024-09-16T13:22:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:22:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:22:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:22:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:22:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:22:55 2024 2024-09-16T13:22:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:22:55 2024 2024-09-16T13:22:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:22:55 2024 2024-09-16T13:22:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:22:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:22:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 7c2f6603-8476-43b9-946c-db51ec3d98a8 2024-09-16T13:22:56.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:22:56.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:22:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary7 2024-09-16T13:22:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd6 2024-09-16T13:22:56.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:22:56.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd6 2024-09-16T13:22:56.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:22:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 528384/2621440 done 2024-09-16T13:22:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:22:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: f61c5870-fad2-4279-b97b-eddc0e7c151e 2024-09-16T13:22:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:22:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:22:57.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:22:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:22:57.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:22:57.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:22:57.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 Snapshot ID: 4640 2024-09-16T13:22:57.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:22:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4641 2024-09-16T13:22:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:22:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T13:22:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:22:57.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary7 2024-09-16T13:22:57.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd6 test-primary7 2024-09-16T13:22:57.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:22:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary7 2024-09-16T13:22:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:22:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-16T13:22:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:22:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:22:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:22:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:22:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:22:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary8 2024-09-16T13:22:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary8 2024-09-16T13:22:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary7 2024-09-16T13:22:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-16T13:22:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:22:57.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:22:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:22:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:22:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:22:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:57.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary7 2024-09-16T13:22:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-16T13:22:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-16T13:22:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:22:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:22:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:22:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:22:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:57.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-16T13:22:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:22:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-16T13:22:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:22:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:22:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-16T13:22:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:22:57.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-16T13:22:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4645 2024-09-16T13:22:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4649 2024-09-16T13:22:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:22:58.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:22:58.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:22:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-16T13:22:58.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:22:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:22:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:22:58.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:22:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary8': 2024-09-16T13:22:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:22:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:22:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:22:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 31975af435fc 2024-09-16T13:22:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.31975af435fc 2024-09-16T13:22:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:22:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:22:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:22:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:22:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:22:57 2024 2024-09-16T13:22:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:22:57 2024 2024-09-16T13:22:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:22:57 2024 2024-09-16T13:22:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:22:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:22:58.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 0be91fa0-3638-4a36-88bc-96f6a810458c 2024-09-16T13:22:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:22:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:22:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary8 2024-09-16T13:22:58.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd7 2024-09-16T13:22:58.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:22:58.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd7 2024-09-16T13:22:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:22:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:22:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:22:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 3254a9c8-7ec1-4384-b4e7-b376ae3ceb77 2024-09-16T13:22:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:22:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:22:58.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:22:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:22:58.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:22:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:22:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:22:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:22:59.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary8 2024-09-16T13:22:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd7 test-primary8 2024-09-16T13:22:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:22:59.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary8 2024-09-16T13:22:59.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:22:59.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:22:59.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:22:59.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-16T13:22:59.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary9 2024-09-16T13:22:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary9 2024-09-16T13:22:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-16T13:22:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary8 2024-09-16T13:22:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:22:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:22:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:59.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary8 2024-09-16T13:22:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:22:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:22:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-16T13:22:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-16T13:22:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:22:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:22:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:22:59.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:22:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:22:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:22:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:22:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:22:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-16T13:22:59.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:22:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:22:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-16T13:22:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:22:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:22:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-16T13:22:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:22:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-16T13:22:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:22:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:22:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:59.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:22:59.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:22:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4653 2024-09-16T13:22:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:22:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:22:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:22:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:22:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:22:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:22:59.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:23:00.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4652 2024-09-16T13:23:00.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:00.591 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T13:23:00.581+0000 7f4071c2f700 -1 reset not still connected to 0x5635a6c75110 2024-09-16T13:23:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:23:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:23:00.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:23:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:23:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:00.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:23:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:23:00.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:23:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-16T13:23:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary9': 2024-09-16T13:23:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:23:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:23:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:23:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 31a82ff7e4de 2024-09-16T13:23:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.31a82ff7e4de 2024-09-16T13:23:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:23:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:23:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:23:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:23:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:22:59 2024 2024-09-16T13:23:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:22:59 2024 2024-09-16T13:23:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:22:59 2024 2024-09-16T13:23:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:23:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:23:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 18f8853c-a255-4064-a250-a3c514f2d73d 2024-09-16T13:23:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:23:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:23:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary9 2024-09-16T13:23:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd8 2024-09-16T13:23:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:23:01.146 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd8 2024-09-16T13:23:01.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:23:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:23:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:23:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: a06e3f35-8641-4b62-af70-fa49f7ab3728 2024-09-16T13:23:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:23:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:23:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:23:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:23:01.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:23:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:01.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:23:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:23:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:23:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:23:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:23:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:23:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): Snapshot ID: 4658 2024-09-16T13:23:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:01.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4657 2024-09-16T13:23:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:01.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T13:23:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:23:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:23:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary9 2024-09-16T13:23:01.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd8 test-primary9 2024-09-16T13:23:02.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:23:02.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary9 2024-09-16T13:23:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:23:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-16T13:23:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:23:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:23:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary10 2024-09-16T13:23:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary10 2024-09-16T13:23:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-16T13:23:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:23:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:23:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:23:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:23:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:23:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:23:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:23:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:23:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:23:02.065 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:23:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-16T13:23:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:23:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:23:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:23:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:23:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:23:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:23:02.067 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:23:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:23:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:23:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary9 2024-09-16T13:23:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-16T13:23:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary9 2024-09-16T13:23:02.068 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-16T13:23:02.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:23:02.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-16T13:23:02.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:23:02.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-16T13:23:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:23:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:23:02.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-16T13:23:02.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4662 2024-09-16T13:23:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:23:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:02.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:23:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:23:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4661 2024-09-16T13:23:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4667 2024-09-16T13:23:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:02.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:23:02.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-16T13:23:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary10': 2024-09-16T13:23:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:23:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:23:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:23:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 31c91e0183e9 2024-09-16T13:23:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.31c91e0183e9 2024-09-16T13:23:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:23:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:23:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:23:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:23:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:23:02 2024 2024-09-16T13:23:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:23:02 2024 2024-09-16T13:23:02.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:23:02 2024 2024-09-16T13:23:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:23:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:23:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: c42e0dcc-72ae-43da-866a-52ef5e032769 2024-09-16T13:23:02.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:23:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:23:02.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary10 2024-09-16T13:23:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd9 2024-09-16T13:23:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:23:03.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd9 2024-09-16T13:23:03.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:23:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:23:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:23:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 019cf4fd-32a0-4349-91c5-89a1b21e9567 2024-09-16T13:23:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:23:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:23:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:23:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:23:03.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:23:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:23:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:23:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:23:03.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): Snapshot ID: 4663 2024-09-16T13:23:03.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4664 2024-09-16T13:23:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:03.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T13:23:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:23:03.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:23:03.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary10 2024-09-16T13:23:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd9 test-primary10 2024-09-16T13:23:04.025 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:23:04.017+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T13:23:04.027 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:23:04.017+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T13:23:04.029 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:23:04.021+0000 7f61b1855700 -1 reset not still connected to 0x557b42003860 2024-09-16T13:23:04.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:23:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary10 2024-09-16T13:23:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:23:04.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-16T13:23:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:23:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:23:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:23:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:23:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:04.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:23:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 109538 2024-09-16T13:23:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:23:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary10 2024-09-16T13:23:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary10 2024-09-16T13:23:04.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-16T13:23:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:23:04.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-16T13:23:04.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4669 2024-09-16T13:23:04.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:23:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:23:04.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:23:04.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4674 2024-09-16T13:23:04.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:23:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:23:04.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:23:05.428 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:23:05.422+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c299bd40 2024-09-16T13:23:05.428 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:23:05.422+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c36a8410 2024-09-16T13:23:05.428 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:23:05.422+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c8c76ea0 2024-09-16T13:23:05.428 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:23:05.422+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8db1d69c0 2024-09-16T13:23:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:23:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:23:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:23:05.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4672 2024-09-16T13:23:05.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:23:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:23:05.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:23:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:23:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:23:05.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:23:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:23:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:23:06.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:23:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4676 2024-09-16T13:23:06.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4678 2024-09-16T13:23:06.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:23:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:23:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:23:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:07.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:23:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:23:07.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:23:07.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4682 2024-09-16T13:23:07.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:07.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4683 2024-09-16T13:23:07.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4684 2024-09-16T13:23:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:23:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:07.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:23:07.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:23:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:08.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4687 2024-09-16T13:23:08.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4690 2024-09-16T13:23:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:08.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:23:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:08.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:23:08.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:23:09.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4693 2024-09-16T13:23:09.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:23:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:09.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:23:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:23:09.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4691 2024-09-16T13:23:09.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:23:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:23:09.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:23:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:23:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:23:10.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:23:10.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4697 2024-09-16T13:23:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:23:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:23:10.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:23:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:23:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:23:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:23:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:23:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:11.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:23:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:23:11.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:11.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:23:11.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:23:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:23:11.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4700 2024-09-16T13:23:11.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4699 2024-09-16T13:23:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:11.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:12.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4704 2024-09-16T13:23:12.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4702 2024-09-16T13:23:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:12.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:12.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4705 2024-09-16T13:23:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:23:12.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:23:12.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:23:12.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:12.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:23:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:23:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:23:12.731 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:23:12.725+0000 7f61b1855700 -1 reset not still connected to 0x557b3ce412b0 2024-09-16T13:23:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:23:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:23:13.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:23:13.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4709 2024-09-16T13:23:13.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4707 2024-09-16T13:23:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:13.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:23:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:23:14.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:23:14.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4714 2024-09-16T13:23:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:23:14.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:23:14.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:23:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4711 2024-09-16T13:23:14.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:14.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:15.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4716 2024-09-16T13:23:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:23:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:15.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:23:15.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:23:15.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:15.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:23:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:23:15.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:23:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:15.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:23:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:23:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:23:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4720 2024-09-16T13:23:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4719 2024-09-16T13:23:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:23:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:23:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:23:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:23:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:16.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:23:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:23:16.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:16.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:17.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4725 2024-09-16T13:23:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:23:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:17.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:23:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:23:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:23:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:23:17.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:23:17.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:17.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4722 2024-09-16T13:23:17.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4723 2024-09-16T13:23:17.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:17.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:18.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4730 2024-09-16T13:23:18.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4729 2024-09-16T13:23:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:23:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:23:18.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:23:19.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4732 2024-09-16T13:23:19.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:19.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4731 2024-09-16T13:23:19.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:23:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:23:19.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:23:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:23:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:19.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:23:19.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:23:20.001 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:23:19.993+0000 7ff4f6946700 -1 reset not still connected to 0x55755cd14270 2024-09-16T13:23:20.001 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:23:19.993+0000 7ff4f6946700 -1 reset not still connected to 0x557560e465b0 2024-09-16T13:23:20.001 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:23:19.993+0000 7ff4f6946700 -1 reset not still connected to 0x5575660744e0 2024-09-16T13:23:20.001 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:23:19.993+0000 7ff4f6946700 -1 reset not still connected to 0x5575754c1450 2024-09-16T13:23:20.001 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:23:19.993+0000 7ff4f6946700 -1 reset not still connected to 0x55758fc0d790 2024-09-16T13:23:20.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4737 2024-09-16T13:23:20.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:23:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:20.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:23:20.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:23:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:23:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:23:20.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:23:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:20.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:23:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:23:20.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:23:21.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4739 2024-09-16T13:23:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:23:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:23:21.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:23:21.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4740 2024-09-16T13:23:21.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:23:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:23:21.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:23:21.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:22.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4743 2024-09-16T13:23:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:23:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:23:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:23:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4745 2024-09-16T13:23:22.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4744 2024-09-16T13:23:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:23:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:22.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:22.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:23:22.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:23:22.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:23:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:23:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:23:23.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4749 2024-09-16T13:23:23.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4750 2024-09-16T13:23:23.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:23.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4753 2024-09-16T13:23:24.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4754 2024-09-16T13:23:24.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:23:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:23:24.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:23:24.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:24.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:23:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:23:24.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:23:25.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4755 2024-09-16T13:23:25.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:23:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:23:25.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:23:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:23:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:23:25.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:23:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:23:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:25.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:23:25.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:23:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4760 2024-09-16T13:23:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:23:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:23:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:23:26.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4759 2024-09-16T13:23:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:23:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:23:26.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:23:26.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:26.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:27.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:27.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:23:27.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:23:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:23:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:23:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:23:27.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:23:27.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4765 2024-09-16T13:23:27.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:27.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4763 2024-09-16T13:23:27.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4764 2024-09-16T13:23:27.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:28.506 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4770 2024-09-16T13:23:28.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4767 2024-09-16T13:23:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:28.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:23:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:23:28.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:23:28.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:23:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:23:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:23:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:23:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:23:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:23:29.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4771 2024-09-16T13:23:29.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4772 2024-09-16T13:23:29.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:29.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:23:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:23:30.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:23:30.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4777 2024-09-16T13:23:30.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:23:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:23:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:23:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:23:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:23:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:23:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:23:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:31.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:23:31.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:23:31.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4778 2024-09-16T13:23:31.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4779 2024-09-16T13:23:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:31.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:32.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4783 2024-09-16T13:23:32.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:32.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:23:32.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:32.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:32.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:23:32.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:23:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:23:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:23:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:23:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4781 2024-09-16T13:23:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4784 2024-09-16T13:23:32.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:32.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:23:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:23:33.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:23:33.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4787 2024-09-16T13:23:33.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4786 2024-09-16T13:23:33.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:33.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:23:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:23:34.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:23:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4793 2024-09-16T13:23:34.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4792 2024-09-16T13:23:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:23:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:23:34.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:23:34.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:34.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:35.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4795 2024-09-16T13:23:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:23:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:23:35.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:23:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:23:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:23:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:23:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:23:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:23:35.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:23:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4799 2024-09-16T13:23:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4800 2024-09-16T13:23:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:23:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:23:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:23:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:23:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:23:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:23:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:36.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:37.594 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4803 2024-09-16T13:23:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:23:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:23:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:23:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:23:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:23:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:23:37.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4804 2024-09-16T13:23:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4805 2024-09-16T13:23:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:37.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:38.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4806 2024-09-16T13:23:38.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4809 2024-09-16T13:23:38.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:23:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:23:38.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:23:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4812 2024-09-16T13:23:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:39.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:23:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:23:39.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:23:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:23:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:23:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:23:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:39.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4811 2024-09-16T13:23:39.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:40.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4817 2024-09-16T13:23:40.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:23:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:23:40.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:23:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:23:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:23:40.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:23:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:23:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:23:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:23:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:23:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:23:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:23:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:23:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:23:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:23:41.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4819 2024-09-16T13:23:41.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4820 2024-09-16T13:23:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:41.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:42.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4824 2024-09-16T13:23:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4823 2024-09-16T13:23:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:23:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:23:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:23:42.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:42.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4825 2024-09-16T13:23:42.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:42.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:23:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:23:42.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:23:43.545 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:23:43.534+0000 7f74def64700 -1 reset not still connected to 0x55945a2ca410 2024-09-16T13:23:43.545 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:23:43.534+0000 7f74def64700 -1 reset not still connected to 0x55945fcd1040 2024-09-16T13:23:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:23:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:23:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:23:43.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4826 2024-09-16T13:23:43.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4827 2024-09-16T13:23:43.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:43.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:44.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4831 2024-09-16T13:23:44.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4833 2024-09-16T13:23:44.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:23:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:23:44.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:23:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:23:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:23:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:23:45.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:23:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:23:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:23:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:23:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:45.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:23:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:23:45.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:23:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:23:45.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:23:45.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4836 2024-09-16T13:23:45.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:23:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:23:46.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:23:46.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4839 2024-09-16T13:23:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:46.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:23:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:23:46.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:23:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4840 2024-09-16T13:23:46.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:46.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:23:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:23:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:23:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4844 2024-09-16T13:23:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:23:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:23:47.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:23:47.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:47.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4845 2024-09-16T13:23:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4841 2024-09-16T13:23:47.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:48.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4848 2024-09-16T13:23:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4849 2024-09-16T13:23:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:48.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:23:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:23:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:23:49.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:23:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:23:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:23:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4853 2024-09-16T13:23:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:23:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:23:49.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:23:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4851 2024-09-16T13:23:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:49.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:50.544 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:23:50.538+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c299aa90 2024-09-16T13:23:50.548 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:23:50.538+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c36a8410 2024-09-16T13:23:50.548 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:23:50.538+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c8c76ea0 2024-09-16T13:23:50.548 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:23:50.538+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8db1d69c0 2024-09-16T13:23:50.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4857 2024-09-16T13:23:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:23:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:23:50.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:23:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:50.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:23:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:23:50.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:23:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:50.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:23:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:23:50.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:23:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:23:51.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:23:51.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:23:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:23:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:23:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:23:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4859 2024-09-16T13:23:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4860 2024-09-16T13:23:51.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:51.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:52.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4862 2024-09-16T13:23:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:52.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4864 2024-09-16T13:23:52.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4865 2024-09-16T13:23:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:52.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:23:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:23:52.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:23:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:52.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:52.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:23:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:23:52.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:23:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:23:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:23:53.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:23:53.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4867 2024-09-16T13:23:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4866 2024-09-16T13:23:53.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:53.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:23:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:23:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:23:54.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4873 2024-09-16T13:23:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:23:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:23:54.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:23:54.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4871 2024-09-16T13:23:54.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:54.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:23:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:23:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:23:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:23:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:23:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:23:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:23:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:55.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:23:55.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:23:55.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4877 2024-09-16T13:23:55.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:56.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4878 2024-09-16T13:23:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:23:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:23:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:23:56.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4880 2024-09-16T13:23:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:23:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:56.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:23:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:23:56.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:56.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:57.535 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:23:57.522+0000 7f74def64700 -1 reset not still connected to 0x559441c00a90 2024-09-16T13:23:57.535 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:23:57.526+0000 7f74def64700 -1 reset not still connected to 0x55945a2ca410 2024-09-16T13:23:57.535 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:23:57.526+0000 7f74def64700 -1 reset not still connected to 0x55945fcd1040 2024-09-16T13:23:57.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:23:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:23:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:23:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:23:57.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:23:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:23:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4884 2024-09-16T13:23:57.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4885 2024-09-16T13:23:57.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:57.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:57.763 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:23:57.752+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0ea0 2024-09-16T13:23:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4886 2024-09-16T13:23:58.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4889 2024-09-16T13:23:58.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:58.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4890 2024-09-16T13:23:58.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:23:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:23:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:23:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4891 2024-09-16T13:23:59.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:59.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4895 2024-09-16T13:23:59.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:23:59.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:23:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:23:59.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:23:59.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:23:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:23:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:23:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:23:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:23:59.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:24:00.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4898 2024-09-16T13:24:00.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:24:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:24:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:24:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:24:00.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:24:00.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:24:01.507 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4901 2024-09-16T13:24:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4900 2024-09-16T13:24:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:01.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:24:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:24:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:24:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:24:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:24:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:24:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:24:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:24:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:24:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:24:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:24:02.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:24:02.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4904 2024-09-16T13:24:02.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:02.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4905 2024-09-16T13:24:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:24:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:24:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:24:02.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:24:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:24:03.505 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:24:03.523 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4908 2024-09-16T13:24:03.532 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4909 2024-09-16T13:24:03.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:03.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:03.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4910 2024-09-16T13:24:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:04.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4915 2024-09-16T13:24:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:24:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:24:04.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:24:04.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:24:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:24:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:24:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4913 2024-09-16T13:24:04.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:24:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:24:05.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:24:05.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4916 2024-09-16T13:24:05.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:24:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:24:05.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:24:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4920 2024-09-16T13:24:06.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:24:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:24:06.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:24:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:24:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:24:06.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:24:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:06.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4921 2024-09-16T13:24:06.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:06.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:24:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:24:06.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:24:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:24:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:24:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:24:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:24:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:24:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:24:07.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4923 2024-09-16T13:24:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4925 2024-09-16T13:24:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:07.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:08.519 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4930 2024-09-16T13:24:08.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4929 2024-09-16T13:24:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:08.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4928 2024-09-16T13:24:08.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:24:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:24:08.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:24:08.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:08.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:09.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4933 2024-09-16T13:24:09.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4931 2024-09-16T13:24:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:24:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:24:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:24:09.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:09.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:24:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:24:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:24:09.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:09.608 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:24:09.602+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c299aa90 2024-09-16T13:24:09.611 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:24:09.602+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c36a8410 2024-09-16T13:24:09.611 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:24:09.602+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c8c76ea0 2024-09-16T13:24:10.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4938 2024-09-16T13:24:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:24:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:24:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:24:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:24:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:24:10.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:24:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:24:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:24:11.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:24:11.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4940 2024-09-16T13:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:24:11.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:11.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:11.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:24:11.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:24:11.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4941 2024-09-16T13:24:11.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:11.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:24:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:12.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:24:12.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:24:12.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:24:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:24:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:24:12.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4944 2024-09-16T13:24:12.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4945 2024-09-16T13:24:12.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:12.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:13.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4949 2024-09-16T13:24:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4948 2024-09-16T13:24:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:24:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:24:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:24:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:13.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4946 2024-09-16T13:24:13.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:14.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4952 2024-09-16T13:24:14.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:14.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:24:14.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:14.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:14.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:24:14.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:24:14.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4953 2024-09-16T13:24:14.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:24:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:24:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:24:14.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:15.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4956 2024-09-16T13:24:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:15.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:24:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:24:15.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:24:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:15.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:24:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:24:15.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:24:15.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:24:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:24:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:24:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4961 2024-09-16T13:24:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:24:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:24:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:24:16.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:24:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:16.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:24:16.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:24:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4960 2024-09-16T13:24:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:17.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4965 2024-09-16T13:24:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:24:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:24:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:24:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:24:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:24:17.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:24:17.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4964 2024-09-16T13:24:17.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4970 2024-09-16T13:24:18.508 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4967 2024-09-16T13:24:18.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4968 2024-09-16T13:24:18.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:18.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:24:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:24:18.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:24:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4975 2024-09-16T13:24:19.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4971 2024-09-16T13:24:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:24:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:24:19.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:24:19.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:19.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:24:19.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:24:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:24:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4978 2024-09-16T13:24:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:24:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:24:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:24:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:24:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:24:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:24:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:24:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:24:21.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:24:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:24:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:24:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:24:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:24:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:24:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:24:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4980 2024-09-16T13:24:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4981 2024-09-16T13:24:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:21.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:24:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:24:22.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:24:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:24:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:22.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:24:22.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:24:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4984 2024-09-16T13:24:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4985 2024-09-16T13:24:22.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:22.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:24:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:24:23.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:24:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4986 2024-09-16T13:24:23.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4988 2024-09-16T13:24:23.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4987 2024-09-16T13:24:23.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:23.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:23.731 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:24:23.722+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e36ab1e0 2024-09-16T13:24:23.734 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:24:23.726+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e5d9d2b0 2024-09-16T13:24:23.734 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:24:23.726+0000 7f315ffe6700 -1 reset not still connected to 0x55c1f81f6d00 2024-09-16T13:24:23.734 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:24:23.726+0000 7f315ffe6700 -1 reset not still connected to 0x55c201562820 2024-09-16T13:24:23.734 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:24:23.726+0000 7f315ffe6700 -1 reset not still connected to 0x55c201563c70 2024-09-16T13:24:24.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4991 2024-09-16T13:24:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:24:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:24:24.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:24:24.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:24:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:24:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:24:24.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4995 2024-09-16T13:24:24.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:24.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:25.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:25.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 4996 2024-09-16T13:24:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:24:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:24:25.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:24:25.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:24:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:24:25.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:24:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:24:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:24:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:24:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5001 2024-09-16T13:24:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:24:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:24:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:24:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:24:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:24:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:24:26.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5000 2024-09-16T13:24:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:26.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:27.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5005 2024-09-16T13:24:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:24:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:24:27.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:24:27.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5004 2024-09-16T13:24:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:24:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:24:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:24:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:28.509 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5008 2024-09-16T13:24:28.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5009 2024-09-16T13:24:28.527 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:28.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5010 2024-09-16T13:24:28.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:28.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:24:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:24:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:24:29.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5015 2024-09-16T13:24:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5014 2024-09-16T13:24:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:29.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:24:29.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:24:29.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:24:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:24:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:24:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:24:30.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5018 2024-09-16T13:24:30.526 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:24:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:24:30.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:24:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:24:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:24:30.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:24:31.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5021 2024-09-16T13:24:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:24:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:24:31.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:24:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:31.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:24:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:24:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:24:31.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5020 2024-09-16T13:24:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:31.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:31.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:31.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:24:31.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:31.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:31.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:24:31.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:24:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:24:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:24:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:24:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:24:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:32.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:24:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:24:32.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5024 2024-09-16T13:24:32.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:32.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5025 2024-09-16T13:24:32.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:24:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:24:33.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:24:33.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5029 2024-09-16T13:24:33.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5030 2024-09-16T13:24:33.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5028 2024-09-16T13:24:33.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:33.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5035 2024-09-16T13:24:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5034 2024-09-16T13:24:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:24:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:24:34.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:24:34.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:24:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:24:34.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:24:34.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5036 2024-09-16T13:24:35.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:24:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:24:35.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:24:35.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:24:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:24:35.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:24:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5040 2024-09-16T13:24:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:24:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:24:36.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:24:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:36.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5041 2024-09-16T13:24:36.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:24:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:24:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:24:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:24:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:36.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:24:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:24:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:24:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:24:37.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:24:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:24:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:24:37.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:24:37.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5044 2024-09-16T13:24:37.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5045 2024-09-16T13:24:37.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:37.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:38.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5049 2024-09-16T13:24:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5048 2024-09-16T13:24:38.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5050 2024-09-16T13:24:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:38.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:38.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:24:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:24:38.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:24:39.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5054 2024-09-16T13:24:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:24:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:24:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:24:39.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5051 2024-09-16T13:24:39.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:24:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:24:39.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:24:39.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5058 2024-09-16T13:24:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:40.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:24:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:24:40.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:24:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:40.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:24:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:24:40.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:24:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:41.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:24:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:24:41.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:24:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:24:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:24:41.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:24:41.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:24:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:24:41.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:24:41.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5061 2024-09-16T13:24:41.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5060 2024-09-16T13:24:41.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:42.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:24:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:24:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:24:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:24:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:24:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:24:42.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5065 2024-09-16T13:24:42.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5064 2024-09-16T13:24:42.620 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:43.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5067 2024-09-16T13:24:43.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:24:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:24:43.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:24:43.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:43.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5069 2024-09-16T13:24:43.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5066 2024-09-16T13:24:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:43.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:24:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:24:44.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:24:44.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5075 2024-09-16T13:24:44.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5074 2024-09-16T13:24:44.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:24:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:24:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:24:44.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:44.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:45.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5076 2024-09-16T13:24:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:24:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:24:45.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:24:45.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:45.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:24:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:24:45.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:24:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5081 2024-09-16T13:24:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:46.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:24:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:24:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:24:46.618 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5080 2024-09-16T13:24:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:24:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:24:46.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:24:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:24:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:24:46.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:24:47.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5084 2024-09-16T13:24:47.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5085 2024-09-16T13:24:47.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:24:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:24:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:24:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:24:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:24:47.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:24:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:47.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5089 2024-09-16T13:24:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5086 2024-09-16T13:24:48.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:48.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5090 2024-09-16T13:24:48.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:48.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:24:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:24:48.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:24:49.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5095 2024-09-16T13:24:49.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5094 2024-09-16T13:24:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:24:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:24:49.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:24:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:49.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:49.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:24:49.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:49.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:49.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:24:49.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:24:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5098 2024-09-16T13:24:50.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:24:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:24:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:24:50.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:24:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:24:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:24:51.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:24:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:24:51.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:24:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:24:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:24:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:24:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:24:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:24:51.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:24:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5101 2024-09-16T13:24:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5100 2024-09-16T13:24:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:24:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:24:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:24:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:24:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:24:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:24:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5104 2024-09-16T13:24:52.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5105 2024-09-16T13:24:52.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5106 2024-09-16T13:24:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:24:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:24:53.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:24:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5108 2024-09-16T13:24:53.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5107 2024-09-16T13:24:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:54.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5112 2024-09-16T13:24:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:24:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:24:54.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:24:54.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5114 2024-09-16T13:24:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:24:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:24:54.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:24:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5116 2024-09-16T13:24:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:24:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:24:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:24:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:24:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:24:55.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:24:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:24:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:24:56.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:24:56.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5120 2024-09-16T13:24:56.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5121 2024-09-16T13:24:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:24:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:24:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:24:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:24:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:24:56.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:24:57.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:24:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:24:57.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:24:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:24:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:24:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:24:57.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5124 2024-09-16T13:24:57.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5125 2024-09-16T13:24:57.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:58.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5128 2024-09-16T13:24:58.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5129 2024-09-16T13:24:58.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5130 2024-09-16T13:24:58.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:58.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:24:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:24:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:24:59.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5134 2024-09-16T13:24:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:24:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:24:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:24:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:24:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:24:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:24:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:24:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:24:59.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:24:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5133 2024-09-16T13:24:59.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:24:59.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:00.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5138 2024-09-16T13:25:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:25:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:25:00.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:25:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:25:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:00.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:25:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:25:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:25:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:25:01.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:25:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:25:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:25:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:25:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:25:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:25:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:25:01.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5141 2024-09-16T13:25:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5140 2024-09-16T13:25:01.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:25:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:25:02.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:25:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:25:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:25:02.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:25:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5144 2024-09-16T13:25:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5145 2024-09-16T13:25:02.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:02.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:25:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:25:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:25:03.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5147 2024-09-16T13:25:03.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5146 2024-09-16T13:25:03.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:03.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5148 2024-09-16T13:25:03.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:25:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:25:04.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:25:04.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5155 2024-09-16T13:25:04.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:04.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:25:04.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:25:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:25:04.721 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:25:04.712+0000 7ff4f6946700 -1 reset not still connected to 0x557560e465b0 2024-09-16T13:25:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:04.722 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:25:04.712+0000 7ff4f6946700 -1 reset not still connected to 0x5575660744e0 2024-09-16T13:25:04.722 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:25:04.712+0000 7ff4f6946700 -1 reset not still connected to 0x5575754c1450 2024-09-16T13:25:04.725 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:25:04.716+0000 7ff4f6946700 -1 reset not still connected to 0x55758fc0d790 2024-09-16T13:25:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5157 2024-09-16T13:25:05.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5156 2024-09-16T13:25:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:25:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:25:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:25:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:25:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:25:05.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:25:05.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:06.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5161 2024-09-16T13:25:06.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5162 2024-09-16T13:25:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:25:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:25:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:25:06.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:25:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:25:06.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:25:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:25:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:25:06.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:25:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:25:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:25:07.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:25:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5166 2024-09-16T13:25:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5165 2024-09-16T13:25:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:07.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:08.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5169 2024-09-16T13:25:08.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5171 2024-09-16T13:25:08.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5170 2024-09-16T13:25:08.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:25:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:25:08.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:25:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:25:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:25:08.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:25:09.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5174 2024-09-16T13:25:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:09.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:25:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:25:09.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:25:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:25:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:09.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:25:09.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:25:10.539 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5178 2024-09-16T13:25:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5177 2024-09-16T13:25:10.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:10.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:25:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:25:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:25:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:25:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:25:10.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:25:11.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5182 2024-09-16T13:25:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:25:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:25:11.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:25:11.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5181 2024-09-16T13:25:11.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:11.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:25:11.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:25:11.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:25:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:25:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:11.592 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:25:11.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:25:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:25:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:25:12.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:25:12.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5186 2024-09-16T13:25:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5185 2024-09-16T13:25:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:12.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:13.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5191 2024-09-16T13:25:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:25:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:25:13.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:25:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:25:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:25:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:25:13.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5190 2024-09-16T13:25:13.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:13.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5189 2024-09-16T13:25:13.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:25:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:25:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:25:14.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5192 2024-09-16T13:25:14.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:25:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 109672 2024-09-16T13:25:14.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:15.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5198 2024-09-16T13:25:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:25:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:25:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:25:15.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:25:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:25:15.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:25:15.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5197 2024-09-16T13:25:15.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5201 2024-09-16T13:25:16.586 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:25:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:25:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:25:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:25:16.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:25:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 109829 2024-09-16T13:25:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:25:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:25:17.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:25:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:25:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:25:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5203 2024-09-16T13:25:17.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:18.538 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5208 2024-09-16T13:25:18.551 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:25:18.543+0000 7ff4f6946700 -1 reset not still connected to 0x5575562dfee0 2024-09-16T13:25:18.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5207 2024-09-16T13:25:18.585 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:18.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:25:18.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 109986 2024-09-16T13:25:18.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5205 2024-09-16T13:25:18.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:18.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:25:18.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:18.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:25:18.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:25:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5212 2024-09-16T13:25:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:25:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:25:19.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:25:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:20.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5214 2024-09-16T13:25:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:25:20.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 110121 2024-09-16T13:25:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:25:21.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 110336 2024-09-16T13:25:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:25:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:25:21.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:25:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:25:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:25:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:25:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5216 2024-09-16T13:25:21.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:25:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:25:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:25:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:23.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:25:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:25:23.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:25:23.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5218 2024-09-16T13:25:23.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5219 2024-09-16T13:25:23.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:23.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:24.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5222 2024-09-16T13:25:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:25:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 110493 2024-09-16T13:25:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:25.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5224 2024-09-16T13:25:25.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:25:26.632 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 110649 2024-09-16T13:25:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:25:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:26.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:25:26.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:25:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:25:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:25:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:25:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:25:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:25:27.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:25:28.529 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5226 2024-09-16T13:25:28.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:25:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 110906 2024-09-16T13:25:29.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5228 2024-09-16T13:25:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:25:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:25:31.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 111084 2024-09-16T13:25:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T13:25:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:25:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 109539 2024-09-16T13:27:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:27:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:27:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:27:45.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 109673 2024-09-16T13:27:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:27:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:27:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:27:47.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 109830 2024-09-16T13:27:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:27:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:27:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:27:49.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 109987 2024-09-16T13:27:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:27:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:27:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:27:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 110122 2024-09-16T13:27:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:27:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:27:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:27:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 110337 2024-09-16T13:27:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:27:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:27:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:27:55.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 110494 2024-09-16T13:27:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:27:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:27:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:27:57.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 110650 2024-09-16T13:27:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:27:59.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:27:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:27:59.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 110907 2024-09-16T13:28:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:28:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:28:02.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:28:02.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 111085 2024-09-16T13:28:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:28:04.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:28:04.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T13:28:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:28:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:28:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:28:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 1 2024-09-16T13:28:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd0 2024-09-16T13:28:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary1 2024-09-16T13:28:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary1 2024-09-16T13:28:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:28:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:28:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary1 2024-09-16T13:28:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:28:04.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 2 2024-09-16T13:28:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd1 2024-09-16T13:28:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary2 2024-09-16T13:28:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary2 2024-09-16T13:28:04.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary2 2024-09-16T13:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 3 2024-09-16T13:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd2 2024-09-16T13:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary3 2024-09-16T13:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary3 2024-09-16T13:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary3 2024-09-16T13:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 4 2024-09-16T13:28:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:28:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:28:04.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd3 2024-09-16T13:28:04.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary4 2024-09-16T13:28:04.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary4 2024-09-16T13:28:04.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:28:04.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary4 2024-09-16T13:28:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:28:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 5 2024-09-16T13:28:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:28:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd4 2024-09-16T13:28:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary5 2024-09-16T13:28:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary5 2024-09-16T13:28:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:28:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary5 2024-09-16T13:28:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:28:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:28:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 7 2024-09-16T13:28:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:28:04.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd6 2024-09-16T13:28:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary7 2024-09-16T13:28:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary7 2024-09-16T13:28:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:28:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:28:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary7 2024-09-16T13:28:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 8 2024-09-16T13:28:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd7 2024-09-16T13:28:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:28:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:28:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary8 2024-09-16T13:28:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary8 2024-09-16T13:28:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:28:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary8 2024-09-16T13:28:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:28:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:28:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 116924 2024-09-16T13:28:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 10 2024-09-16T13:28:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 6 2024-09-16T13:28:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd9 2024-09-16T13:28:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd5 2024-09-16T13:28:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary10 2024-09-16T13:28:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary10 2024-09-16T13:28:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:28:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary10 2024-09-16T13:28:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary6 2024-09-16T13:28:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary6 2024-09-16T13:28:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:28:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary6 2024-09-16T13:28:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 9 2024-09-16T13:28:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd8 2024-09-16T13:28:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary9 2024-09-16T13:28:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary9 2024-09-16T13:28:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:28:04.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary9 2024-09-16T13:28:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd1 2024-09-16T13:28:04.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:28:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd6 2024-09-16T13:28:04.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:28:04.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd0 2024-09-16T13:28:04.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:28:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:28:04.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd5 2024-09-16T13:28:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd9 2024-09-16T13:28:04.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:28:04.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:28:04.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd2 2024-09-16T13:28:04.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:28:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd3 2024-09-16T13:28:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd8 2024-09-16T13:28:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:28:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:28:04.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd4 2024-09-16T13:28:04.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd7 2024-09-16T13:28:04.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:29:00.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=66f0fb21a19308746378d8f94f6bbf0c 2024-09-16T13:29:00.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary2 2024-09-16T13:29:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=eedc1125b425408d84a4c5a168f27fc1 2024-09-16T13:29:01.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary7 2024-09-16T13:29:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=40326a84d8a7f5dbff4c2ae784e7b6ff 2024-09-16T13:29:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary8 2024-09-16T13:29:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=826d37228277ba040cfa676eb8087955 2024-09-16T13:29:01.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary1 2024-09-16T13:29:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=630b18590e7ee23f9005853cf442bd1a 2024-09-16T13:29:01.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary5 2024-09-16T13:29:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=db048d7d37e1c6b64c9f06be8acafa89 2024-09-16T13:29:01.531 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary6 2024-09-16T13:29:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=77afb3dde41e5946b77c75f36e73d56d 2024-09-16T13:29:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary9 2024-09-16T13:29:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=5df1dfb7a0a5f49cc7d2d0dfed001d5d 2024-09-16T13:29:01.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary4 2024-09-16T13:29:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-16T13:29:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:29:02.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-16T13:29:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-16T13:29:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:29:02.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-16T13:29:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=8574655b74822b08203e171bef00dd1c 2024-09-16T13:29:02.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary3 2024-09-16T13:29:02.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-16T13:29:02.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:02.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:02.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:29:02.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-16T13:29:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-16T13:29:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:29:02.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-16T13:29:02.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-16T13:29:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:29:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-16T13:29:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-16T13:29:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:29:02.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-16T13:29:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-16T13:29:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:29:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-16T13:29:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-16T13:29:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:29:02.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-16T13:29:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-16T13:29:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:29:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-16T13:29:02.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=8283a879298406745131814c5ee3f697 2024-09-16T13:29:02.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary10 2024-09-16T13:29:03.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-16T13:29:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:29:03.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-16T13:29:03.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:29:03.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:29:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:29:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:29:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:29:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:29:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-16T13:29:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-16T13:29:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:29:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:29:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:03.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:29:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:29:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:29:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:29:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-16T13:29:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:29:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:29:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:29:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-16T13:29:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:29:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:29:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:29:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-16T13:29:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-16T13:29:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:29:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:29:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:29:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:29:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:29:03.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:29:04.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:29:04.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:29:04.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:29:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-16T13:29:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:04.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:29:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:29:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:29:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-16T13:29:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:29:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:29:04.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:29:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-16T13:29:04.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:29:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:29:04.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:29:04.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:29:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-16T13:29:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:29:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:04.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:29:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:29:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-16T13:29:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:29:05.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-16T13:29:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:29:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T13:29:05.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-16T13:29:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-16T13:29:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:29:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:29:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T13:29:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-16T13:29:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:29:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T13:29:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T13:29:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T13:29:05.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-16T13:29:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:29:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:05.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:05.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:05.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T13:29:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:29:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b48e7af7-8636-448d-b1d6-46b0539ab6e9 2024-09-16T13:29:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:29:05.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:29:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:03 2024-09-16T13:29:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:29:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:29:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:29:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2709504.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493117,"remote_snapshot_timestamp":1726493117,"replay_state":"idle"} 2024-09-16T13:29:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:28:53' 2024-09-16T13:29:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:29:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:29:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 0be91fa0-3638-4a36-88bc-96f6a810458c 2024-09-16T13:29:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:29:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:29:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:03 2024-09-16T13:29:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:29:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:29:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:29:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2160640.0,"last_snapshot_bytes":2592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493124,"remote_snapshot_timestamp":1726493124,"replay_state":"idle"} 2024-09-16T13:29:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:28:53' 2024-09-16T13:29:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:29:05.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: ab41f43b-2d91-4d86-a967-20ff72553573 2024-09-16T13:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:03 2024-09-16T13:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1693696.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493110,"remote_snapshot_timestamp":1726493110,"replay_state":"idle"} 2024-09-16T13:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:28:53' 2024-09-16T13:29:05.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:29:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:29:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 8dbcffb5-dab3-446c-89d2-a6bb77d01c73 2024-09-16T13:29:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:29:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:29:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:03 2024-09-16T13:29:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:29:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:29:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:29:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4698112.0,"last_snapshot_bytes":2318336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493112,"remote_snapshot_timestamp":1726493112,"replay_state":"idle"} 2024-09-16T13:29:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:28:53' 2024-09-16T13:29:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:29:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:29:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 7c2f6603-8476-43b9-946c-db51ec3d98a8 2024-09-16T13:29:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:29:05.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:29:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:03 2024-09-16T13:29:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:29:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:29:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:29:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":180224.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493122,"remote_snapshot_timestamp":1726493122,"replay_state":"idle"} 2024-09-16T13:29:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:28:53' 2024-09-16T13:29:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:29:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:29:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 64ea0105-23d9-444f-a0cb-021f5706b8b1 2024-09-16T13:29:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:29:05.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:29:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:03 2024-09-16T13:29:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:29:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:29:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:29:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2160640.0,"last_snapshot_bytes":2592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493120,"remote_snapshot_timestamp":1726493120,"replay_state":"idle"} 2024-09-16T13:29:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:28:53' 2024-09-16T13:29:05.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:29:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:29:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:29:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:29:05.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:29:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:29:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:29:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:29:05.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:29:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:29:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:29:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:29:05.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:29:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:29:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:29:05.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:29:05.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:29:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:29:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:05.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:29:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:29:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:29:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:29:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:29:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:29:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:29:05.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:29:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:29:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:29:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:29:05.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:05.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:29:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:29:05.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:29:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:29:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:29:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:29:05.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:29:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:05.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:29:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:05.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:29:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:29:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:29:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:29:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:29:05.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:29:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:29:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:29:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:29:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:29:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:29:05.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:29:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:29:05.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:29:05.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:29:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:29:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:29:05.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:29:05.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:03' 2024-09-16T13:29:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-16 13:29:03' 2024-09-16T13:29:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:29:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:29:03' 2024-09-16T13:29:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:05.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:03' 2024-09-16T13:29:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-16 13:29:03' 2024-09-16T13:29:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:29:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:29:03' 2024-09-16T13:29:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:05.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:29:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:03' 2024-09-16T13:29:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-16 13:29:03' 2024-09-16T13:29:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:29:05.285 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:29:03' 2024-09-16T13:29:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:03' 2024-09-16T13:29:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-16 13:29:03' 2024-09-16T13:29:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:29:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:29:03' 2024-09-16T13:29:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:29:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:29:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:03' 2024-09-16T13:29:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-16 13:29:03' 2024-09-16T13:29:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:29:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:29:03' 2024-09-16T13:29:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:29:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:05.289 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:29:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:03' 2024-09-16T13:29:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-16 13:29:03' 2024-09-16T13:29:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:05.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:29:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:29:03' 2024-09-16T13:29:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:05.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:29:05.486 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:29:05.477+0000 7ff4f6946700 -1 reset not still connected to 0x55755c8aeea0 2024-09-16T13:29:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1693696.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493110,"remote_snapshot_timestamp":1726493110,"replay_state":"idle"}2024-09-16 13:28:53image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2140160.0,"last_snapshot_bytes":2609152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493128,"remote_snapshot_timestamp":1726493128,"replay_state":"idle"}2024-09-16 13:28:53image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4698112.0,"last_snapshot_bytes":2318336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493112,"remote_snapshot_timestamp":1726493112,"replay_state":"idle"}2024-09-16 13:28:53image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2142208.0,"last_snapshot_bytes":2588672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493114,"remote_snapshot_timestamp":1726493114,"replay_state":"idle"}2024-09-16 13:28:53image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":5300224.0,"last_snapshot_bytes":6823936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493116,"remote_snapshot_timestamp":1726493116,"replay_state":"idle"}2024-09-16 13:28:53image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2709504.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493117,"remote_snapshot_timestamp":1726493117,"replay_state":"idle"}2024-09-16 13:28:53image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2160640.0,"last_snapshot_bytes":2592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493120,"remote_snapshot_timestamp":1726493120,"replay_state":"idle"}2024-09-16 13:28:53image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":180224.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493122,"remote_snapshot_timestamp":1726493122,"replay_state":"idle"}2024-09-16 13:28:53image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2160640.0,"last_snapshot_bytes":2592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493124,"remote_snapshot_timestamp":1726493124,"replay_state":"idle"}2024-09-16 13:28:53image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2140160.0,"last_snapshot_bytes":2650112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493127,"remote_snapshot_timestamp":1726493127,"replay_state":"idle"}2024-09-16 13:28:53' 2024-09-16T13:29:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:29:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T13:29:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1693696.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493110,"remote_snapshot_timestamp":1726493110,"replay_state":"idle"}2024-09-16 13:28:53image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2140160.0,"last_snapshot_bytes":2609152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493128,"remote_snapshot_timestamp":1726493128,"replay_state":"idle"}2024-09-16 13:28:53image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4698112.0,"last_snapshot_bytes":2318336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493112,"remote_snapshot_timestamp":1726493112,"replay_state":"idle"}2024-09-16 13:28:53image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2142208.0,"last_snapshot_bytes":2588672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493114,"remote_snapshot_timestamp":1726493114,"replay_state":"idle"}2024-09-16 13:28:53image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":5300224.0,"last_snapshot_bytes":6823936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493116,"remote_snapshot_timestamp":1726493116,"replay_state":"idle"}2024-09-16 13:28:53image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2709504.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493117,"remote_snapshot_timestamp":1726493117,"replay_state":"idle"}2024-09-16 13:28:53image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2160640.0,"last_snapshot_bytes":2592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493120,"remote_snapshot_timestamp":1726493120,"replay_state":"idle"}2024-09-16 13:28:53image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":180224.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493122,"remote_snapshot_timestamp":1726493122,"replay_state":"idle"}2024-09-16 13:28:53image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2160640.0,"last_snapshot_bytes":2592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493124,"remote_snapshot_timestamp":1726493124,"replay_state":"idle"}2024-09-16 13:28:53image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2140160.0,"last_snapshot_bytes":2650112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493127,"remote_snapshot_timestamp":1726493127,"replay_state":"idle"}2024-09-16 13:28:53' 2024-09-16T13:29:05.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:29:05.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T13:29:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:03' 2024-09-16T13:29:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:03' 2024-09-16T13:29:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T13:29:05.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T13:29:05.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1693696.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493110,"remote_snapshot_timestamp":1726493110,"replay_state":"idle"}2024-09-16 13:28:53image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2140160.0,"last_snapshot_bytes":2609152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493128,"remote_snapshot_timestamp":1726493128,"replay_state":"idle"}2024-09-16 13:28:53image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4698112.0,"last_snapshot_bytes":2318336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493112,"remote_snapshot_timestamp":1726493112,"replay_state":"idle"}2024-09-16 13:28:53image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2142208.0,"last_snapshot_bytes":2588672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493114,"remote_snapshot_timestamp":1726493114,"replay_state":"idle"}2024-09-16 13:28:53image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":5300224.0,"last_snapshot_bytes":6823936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493116,"remote_snapshot_timestamp":1726493116,"replay_state":"idle"}2024-09-16 13:28:53image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2709504.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493117,"remote_snapshot_timestamp":1726493117,"replay_state":"idle"}2024-09-16 13:28:53image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2160640.0,"last_snapshot_bytes":2592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493120,"remote_snapshot_timestamp":1726493120,"replay_state":"idle"}2024-09-16 13:28:53image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":180224.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493122,"remote_snapshot_timestamp":1726493122,"replay_state":"idle"}2024-09-16 13:28:53image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2160640.0,"last_snapshot_bytes":2592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493124,"remote_snapshot_timestamp":1726493124,"replay_state":"idle"}2024-09-16 13:28:53image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2140160.0,"last_snapshot_bytes":2650112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493127,"remote_snapshot_timestamp":1726493127,"replay_state":"idle"}2024-09-16 13:28:53' 2024-09-16T13:29:05.595 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:29:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:29:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:05.597 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:29:05.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-16T13:29:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:29:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:29:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:29:05.599 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T13:29:05.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:29:05.600 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:05.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:29:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:05.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:05.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:05.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-16T13:29:05.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:05.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:29:05.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:29:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:29:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1693696.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493110,"remote_snapshot_timestamp":1726493110,"replay_state":"idle"}2024-09-16 13:28:53image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2140160.0,"last_snapshot_bytes":2609152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493128,"remote_snapshot_timestamp":1726493128,"replay_state":"idle"}2024-09-16 13:28:53image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4698112.0,"last_snapshot_bytes":2318336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493112,"remote_snapshot_timestamp":1726493112,"replay_state":"idle"}2024-09-16 13:28:53image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2142208.0,"last_snapshot_bytes":2588672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493114,"remote_snapshot_timestamp":1726493114,"replay_state":"idle"}2024-09-16 13:28:53image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":5300224.0,"last_snapshot_bytes":6823936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493116,"remote_snapshot_timestamp":1726493116,"replay_state":"idle"}2024-09-16 13:28:53image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2709504.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493117,"remote_snapshot_timestamp":1726493117,"replay_state":"idle"}2024-09-16 13:28:53image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2160640.0,"last_snapshot_bytes":2592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493120,"remote_snapshot_timestamp":1726493120,"replay_state":"idle"}2024-09-16 13:28:53image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":180224.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493122,"remote_snapshot_timestamp":1726493122,"replay_state":"idle"}2024-09-16 13:28:53image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2160640.0,"last_snapshot_bytes":2592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493124,"remote_snapshot_timestamp":1726493124,"replay_state":"idle"}2024-09-16 13:28:53image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2140160.0,"last_snapshot_bytes":2650112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493127,"remote_snapshot_timestamp":1726493127,"replay_state":"idle"}2024-09-16 13:28:53' 2024-09-16T13:29:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:29:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T13:29:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1693696.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493110,"remote_snapshot_timestamp":1726493110,"replay_state":"idle"}2024-09-16 13:28:53image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2140160.0,"last_snapshot_bytes":2609152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493128,"remote_snapshot_timestamp":1726493128,"replay_state":"idle"}2024-09-16 13:28:53image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4698112.0,"last_snapshot_bytes":2318336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493112,"remote_snapshot_timestamp":1726493112,"replay_state":"idle"}2024-09-16 13:28:53image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2142208.0,"last_snapshot_bytes":2588672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493114,"remote_snapshot_timestamp":1726493114,"replay_state":"idle"}2024-09-16 13:28:53image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":5300224.0,"last_snapshot_bytes":6823936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493116,"remote_snapshot_timestamp":1726493116,"replay_state":"idle"}2024-09-16 13:28:53image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2709504.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493117,"remote_snapshot_timestamp":1726493117,"replay_state":"idle"}2024-09-16 13:28:53image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2160640.0,"last_snapshot_bytes":2592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493120,"remote_snapshot_timestamp":1726493120,"replay_state":"idle"}2024-09-16 13:28:53image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":180224.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493122,"remote_snapshot_timestamp":1726493122,"replay_state":"idle"}2024-09-16 13:28:53image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2160640.0,"last_snapshot_bytes":2592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493124,"remote_snapshot_timestamp":1726493124,"replay_state":"idle"}2024-09-16 13:28:53image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2140160.0,"last_snapshot_bytes":2650112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493127,"remote_snapshot_timestamp":1726493127,"replay_state":"idle"}2024-09-16 13:28:53' 2024-09-16T13:29:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:03' 2024-09-16T13:29:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:29:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T13:29:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T13:29:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:03' 2024-09-16T13:29:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:29:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T13:29:05.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:29:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:03' 2024-09-16T13:29:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T13:29:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:05.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-16T13:29:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:29:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:29:05.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:29:05.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:29:05.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:05.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:29:05.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:29:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:29:05.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-16T13:29:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:29:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:29:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:29:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-16T13:29:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:29:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:29:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:29:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1693696.0,"last_snapshot_bytes":1699840,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493110,"remote_snapshot_timestamp":1726493110,"replay_state":"idle"}2024-09-16 13:28:53image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2140160.0,"last_snapshot_bytes":2609152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493128,"remote_snapshot_timestamp":1726493128,"replay_state":"idle"}2024-09-16 13:28:53image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4698112.0,"last_snapshot_bytes":2318336,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493112,"remote_snapshot_timestamp":1726493112,"replay_state":"idle"}2024-09-16 13:28:53image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2142208.0,"last_snapshot_bytes":2588672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493114,"remote_snapshot_timestamp":1726493114,"replay_state":"idle"}2024-09-16 13:28:53image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":5300224.0,"last_snapshot_bytes":6823936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493116,"remote_snapshot_timestamp":1726493116,"replay_state":"idle"}2024-09-16 13:28:53image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2709504.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493117,"remote_snapshot_timestamp":1726493117,"replay_state":"idle"}2024-09-16 13:28:53image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2160640.0,"last_snapshot_bytes":2592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493120,"remote_snapshot_timestamp":1726493120,"replay_state":"idle"}2024-09-16 13:28:53image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":180224.0,"last_snapshot_bytes":135168,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493122,"remote_snapshot_timestamp":1726493122,"replay_state":"idle"}2024-09-16 13:28:53image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2160640.0,"last_snapshot_bytes":2592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493124,"remote_snapshot_timestamp":1726493124,"replay_state":"idle"}2024-09-16 13:28:53image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2140160.0,"last_snapshot_bytes":2650112,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493127,"remote_snapshot_timestamp":1726493127,"replay_state":"idle"}2024-09-16 13:28:53' 2024-09-16T13:29:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:29:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T13:29:05.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:03' 2024-09-16T13:29:05.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T13:29:05.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:29:05.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:29:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:05.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-16T13:29:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:29:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:29:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:29:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-16T13:29:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:29:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-16T13:29:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:29:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:05.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:05.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:05.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:05.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T13:29:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:05.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T13:29:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-16T13:29:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:29:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:05.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T13:29:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-16T13:29:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:29:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:05.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T13:29:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:29:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a2ea9fe0-ce1d-400a-8b01-1f096f691b5c 2024-09-16T13:29:05.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:29:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:29:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:03 2024-09-16T13:29:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:29:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:29:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:29:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:05' 2024-09-16T13:29:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:29:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:29:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: c42e0dcc-72ae-43da-866a-52ef5e032769 2024-09-16T13:29:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:05.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:29:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:29:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:03 2024-09-16T13:29:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:29:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:29:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:29:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:05' 2024-09-16T13:29:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:29:05.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:29:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:29:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:29:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:29:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:29:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 18f8853c-a255-4064-a250-a3c514f2d73d 2024-09-16T13:29:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:05.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:29:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:29:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:03 2024-09-16T13:29:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:29:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:29:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:29:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:05' 2024-09-16T13:29:05.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:29:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:29:05.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:29:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:29:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:29:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:29:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: da5b1f11-8a62-48f0-8749-2008dd7a75cc 2024-09-16T13:29:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:05.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:29:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:29:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:03 2024-09-16T13:29:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:29:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:29:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:29:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:05' 2024-09-16T13:29:05.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:29:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:29:05.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:29:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:29:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:29:05.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:29:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:29:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:29:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:29:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:29:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:29:05.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:29:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:05.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:29:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:29:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:29:05.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:29:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:29:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:29:05.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:29:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:29:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:29:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:29:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:29:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:29:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:29:05.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:29:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:03' 2024-09-16T13:29:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-16 13:29:03' 2024-09-16T13:29:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:29:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:29:03' 2024-09-16T13:29:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:03' 2024-09-16T13:29:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-16 13:29:03' 2024-09-16T13:29:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:29:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:29:03' 2024-09-16T13:29:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:03' 2024-09-16T13:29:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:29:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-16 13:29:03' 2024-09-16T13:29:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:29:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:29:03' 2024-09-16T13:29:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:29:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:29:05.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:29:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:29:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:29:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:03' 2024-09-16T13:29:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-16 13:29:03' 2024-09-16T13:29:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:29:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:05.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:29:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:29:03' 2024-09-16T13:29:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:05.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:29:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05' 2024-09-16T13:29:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:29:06.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T13:29:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:03' 2024-09-16T13:29:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T13:29:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:29:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05' 2024-09-16T13:29:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:29:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:29:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T13:29:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-16T13:29:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:29:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:29:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:29:06.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05' 2024-09-16T13:29:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:29:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:03' 2024-09-16T13:29:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:29:03cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image demoted2024-09-16 13:29:05' 2024-09-16T13:29:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:29:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T13:29:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T13:29:06.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T13:29:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:03' 2024-09-16T13:29:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:03' 2024-09-16T13:29:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T13:29:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T13:29:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:29:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:29:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-16T13:29:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:29:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:29:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:29:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:29:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:29:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:29:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:29:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:06.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:06.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:06.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-16T13:29:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:29:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:29:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:29:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-16T13:29:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:29:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:29:06.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:29:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-16T13:29:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:29:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:06.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:06.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:06.602 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-16T13:29:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-16T13:29:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:29:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:06.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-16T13:29:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-16T13:29:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:29:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:06.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-16T13:29:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:29:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-16T13:29:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:06.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-16T13:29:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-16T13:29:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:29:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:06.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:06.622 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-16T13:29:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:29:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b48e7af7-8636-448d-b1d6-46b0539ab6e9 2024-09-16T13:29:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:29:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:29:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:05 2024-09-16T13:29:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:29:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:29:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:29:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:03' 2024-09-16T13:29:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:29:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:29:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:29:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:29:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 64ea0105-23d9-444f-a0cb-021f5706b8b1 2024-09-16T13:29:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:29:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:29:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:05 2024-09-16T13:29:06.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:29:06.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:29:06.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:06.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:29:06.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:03' 2024-09-16T13:29:06.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:29:06.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:29:06.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:29:06.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:29:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:29:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:29:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:29:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:29:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:29:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:29:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:29:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:29:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:29:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:29:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:29:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:29:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:29:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:05' 2024-09-16T13:29:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-16 13:29:05' 2024-09-16T13:29:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:29:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:29:05' 2024-09-16T13:29:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:29:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:05' 2024-09-16T13:29:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-16 13:29:05' 2024-09-16T13:29:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:29:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:29:05' 2024-09-16T13:29:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:29:06.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-16T13:29:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:29:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:29:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 8dbcffb5-dab3-446c-89d2-a6bb77d01c73 2024-09-16T13:29:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:29:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:29:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:05 2024-09-16T13:29:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:29:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:29:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:29:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:03' 2024-09-16T13:29:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:29:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:29:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: ab41f43b-2d91-4d86-a967-20ff72553573 2024-09-16T13:29:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:29:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:29:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:05 2024-09-16T13:29:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:29:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:29:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:29:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:03' 2024-09-16T13:29:06.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:29:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:29:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:29:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:29:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:29:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:06.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:29:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:29:06.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:29:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:29:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:29:06.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:29:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:29:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:29:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:29:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:29:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:29:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 7c2f6603-8476-43b9-946c-db51ec3d98a8 2024-09-16T13:29:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:29:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:29:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:05 2024-09-16T13:29:06.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:29:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:29:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:29:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:03' 2024-09-16T13:29:06.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:29:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:29:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-16T13:29:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:29:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:29:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:05' 2024-09-16T13:29:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-16 13:29:05' 2024-09-16T13:29:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:29:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:29:05' 2024-09-16T13:29:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:29:06.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:29:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:29:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:29:06.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:29:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:29:06.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:29:06.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:29:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:05' 2024-09-16T13:29:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-16 13:29:05' 2024-09-16T13:29:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:29:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:29:05' 2024-09-16T13:29:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:29:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:29:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:29:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:29:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:05' 2024-09-16T13:29:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-16 13:29:05' 2024-09-16T13:29:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:29:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:29:05' 2024-09-16T13:29:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:06.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:29:06.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:29:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 0be91fa0-3638-4a36-88bc-96f6a810458c 2024-09-16T13:29:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:29:06.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:29:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:05 2024-09-16T13:29:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:29:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:29:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:29:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:03' 2024-09-16T13:29:06.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:29:06.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:29:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:29:06.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:29:06.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:29:06.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:29:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:06.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:29:06.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:29:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:29:06.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:29:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:05' 2024-09-16T13:29:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-16 13:29:05' 2024-09-16T13:29:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:29:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:29:05' 2024-09-16T13:29:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:06.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:29:06.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03' 2024-09-16T13:29:06.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:29:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03' 2024-09-16T13:29:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:29:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T13:29:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T13:29:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:05' 2024-09-16T13:29:06.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T13:29:06.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:05' 2024-09-16T13:29:06.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T13:29:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:29:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:29:06.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:29:06.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-16T13:29:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:29:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:29:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:29:06.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-16T13:29:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:06.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-16T13:29:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:29:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:29:06.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-16T13:29:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03' 2024-09-16T13:29:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:29:06.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T13:29:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:05' 2024-09-16T13:29:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T13:29:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03' 2024-09-16T13:29:06.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:29:06.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T13:29:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03' 2024-09-16T13:29:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:29:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:05' 2024-09-16T13:29:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:29:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T13:29:06.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T13:29:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:29:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:07.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:29:07.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:07.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:29:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:07.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-16T13:29:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:29:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:29:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-16T13:29:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:05' 2024-09-16T13:29:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-16T13:29:07.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:29:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:29:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-16T13:29:07.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T13:29:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:29:07.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:07.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:29:07.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:07.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:07.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:07.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-16T13:29:07.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:29:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:29:07.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-16T13:29:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03' 2024-09-16T13:29:07.050 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:29:07.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T13:29:07.062 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:05' 2024-09-16T13:29:07.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T13:29:07.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:29:07.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:07.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:29:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:07.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-16T13:29:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:29:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:29:07.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-16T13:29:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-16T13:29:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:29:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:07.184 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-16T13:29:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-16T13:29:07.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:29:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:07.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:07.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-16T13:29:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-16T13:29:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:29:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:07.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:07.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-16T13:29:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:29:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:29:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-16T13:29:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:29:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:07.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-16T13:29:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:29:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 18f8853c-a255-4064-a250-a3c514f2d73d 2024-09-16T13:29:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:29:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:29:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:05 2024-09-16T13:29:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:29:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:29:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:29:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:03' 2024-09-16T13:29:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:29:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:29:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:29:07.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:29:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:29:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:07.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:29:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:29:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:29:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:29:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:29:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:05' 2024-09-16T13:29:07.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-16 13:29:05' 2024-09-16T13:29:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:29:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:29:05' 2024-09-16T13:29:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:07.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:29:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:29:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: da5b1f11-8a62-48f0-8749-2008dd7a75cc 2024-09-16T13:29:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:29:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:29:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:05 2024-09-16T13:29:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:29:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:29:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:29:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:03' 2024-09-16T13:29:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:29:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:29:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:29:07.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:29:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:29:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:07.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:29:07.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:29:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: c42e0dcc-72ae-43da-866a-52ef5e032769 2024-09-16T13:29:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:29:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:29:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:05 2024-09-16T13:29:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:29:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:29:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:29:07.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:03' 2024-09-16T13:29:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:29:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:29:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:29:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a2ea9fe0-ce1d-400a-8b01-1f096f691b5c 2024-09-16T13:29:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:29:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:29:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:05 2024-09-16T13:29:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:29:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:29:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:29:07.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:29:07.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:29:03' 2024-09-16T13:29:07.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:29:07.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:29:07.268 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:29:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:29:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:29:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:29:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:29:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:29:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:29:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:29:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:29:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image demoted 2024-09-16T13:29:07.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:29:07.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:29:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:29:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:05' 2024-09-16T13:29:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-16 13:29:05' 2024-09-16T13:29:07.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:29:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:29:05' 2024-09-16T13:29:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:07.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:29:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:29:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:29:07.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:29:07.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:29:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:29:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:29:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:29:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:05' 2024-09-16T13:29:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-16 13:29:05' 2024-09-16T13:29:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:05' 2024-09-16T13:29:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:29:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:29:05' 2024-09-16T13:29:07.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-16 13:29:05' 2024-09-16T13:29:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:29:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:29:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:29:05' 2024-09-16T13:29:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:29:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:29:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:29:07.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:29:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03' 2024-09-16T13:29:07.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:29:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T13:29:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03' 2024-09-16T13:29:07.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:29:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T13:29:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:05' 2024-09-16T13:29:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03' 2024-09-16T13:29:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:29:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T13:29:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T13:29:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:05' 2024-09-16T13:29:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T13:29:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:29:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:29:07.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:05' 2024-09-16T13:29:07.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T13:29:07.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:07.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:29:07.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:07.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-16T13:29:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:29:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:29:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-16T13:29:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:29:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:29:07.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:07.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:29:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+unknownremote image demoted42654268mirror.0smithi1722024-09-16 13:29:05cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:29:03' 2024-09-16T13:29:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:29:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-16T13:29:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:07.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:29:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:29:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-16T13:29:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T13:29:07.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-16T13:29:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:29:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:29:07.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-16T13:29:07.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:29:05' 2024-09-16T13:29:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T13:29:07.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:29:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:29:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:29:07.497 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:29:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:29:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-16T13:29:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:29:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:29:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:29:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:29:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-16T13:29:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:29:07.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:29:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:29:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:29:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:29:07.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:29:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:29:07.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:29:07.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:29:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:29:07.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:29:07.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary5 2024-09-16T13:29:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:29:07.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary8 2024-09-16T13:29:07.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:29:07.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary3 2024-09-16T13:29:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:29:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary10 2024-09-16T13:29:07.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:29:07.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary6 2024-09-16T13:29:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:29:07.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary1 2024-09-16T13:29:07.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:29:07.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary2 2024-09-16T13:29:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:29:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary9 2024-09-16T13:29:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:29:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary7 2024-09-16T13:29:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:29:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary4 2024-09-16T13:29:07.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd0 2024-09-16T13:29:07.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:29:07.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd0 2024-09-16T13:29:07.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd1 2024-09-16T13:29:07.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:29:07.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd1 2024-09-16T13:29:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd2 2024-09-16T13:29:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:29:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd2 2024-09-16T13:29:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd3 2024-09-16T13:29:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:29:08.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd3 2024-09-16T13:29:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd4 2024-09-16T13:29:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd4 2024-09-16T13:29:08.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:29:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd5 2024-09-16T13:29:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd5 2024-09-16T13:29:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:29:08.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd6 2024-09-16T13:29:08.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:29:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd6 2024-09-16T13:29:08.203 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd7 2024-09-16T13:29:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd7 2024-09-16T13:29:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:29:08.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd8 2024-09-16T13:29:08.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:29:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd8 2024-09-16T13:29:08.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd9 2024-09-16T13:29:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:29:08.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd9 2024-09-16T13:30:43.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=8574655b74822b08203e171bef00dd1c 2024-09-16T13:30:43.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd6 2024-09-16T13:30:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=66f0fb21a19308746378d8f94f6bbf0c 2024-09-16T13:30:44.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd3 2024-09-16T13:30:44.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 8574655b74822b08203e171bef00dd1c != \8\5\7\4\6\5\5\b\7\4\8\2\2\b\0\8\2\0\3\e\1\7\1\b\e\f\0\0\d\d\1\c ]] 2024-09-16T13:30:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=eedc1125b425408d84a4c5a168f27fc1 2024-09-16T13:30:45.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd7 2024-09-16T13:30:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=db048d7d37e1c6b64c9f06be8acafa89 2024-09-16T13:30:45.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd5 2024-09-16T13:30:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=8283a879298406745131814c5ee3f697 2024-09-16T13:30:45.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd2 2024-09-16T13:30:45.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=77afb3dde41e5946b77c75f36e73d56d 2024-09-16T13:30:45.411 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd9 2024-09-16T13:30:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=826d37228277ba040cfa676eb8087955 2024-09-16T13:30:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd4 2024-09-16T13:30:45.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 66f0fb21a19308746378d8f94f6bbf0c != \6\6\f\0\f\b\2\1\a\1\9\3\0\8\7\4\6\3\7\8\d\8\f\9\4\f\6\b\b\f\0\c ]] 2024-09-16T13:30:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ eedc1125b425408d84a4c5a168f27fc1 != \e\e\d\c\1\1\2\5\b\4\2\5\4\0\8\d\8\4\a\4\c\5\a\1\6\8\f\2\7\f\c\1 ]] 2024-09-16T13:30:46.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=5df1dfb7a0a5f49cc7d2d0dfed001d5d 2024-09-16T13:30:46.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd8 2024-09-16T13:30:46.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ db048d7d37e1c6b64c9f06be8acafa89 != \d\b\0\4\8\d\7\d\3\7\e\1\c\6\b\6\4\c\9\f\0\6\b\e\8\a\c\a\f\a\8\9 ]] 2024-09-16T13:30:46.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 8283a879298406745131814c5ee3f697 != \8\2\8\3\a\8\7\9\2\9\8\4\0\6\7\4\5\1\3\1\8\1\4\c\5\e\e\3\f\6\9\7 ]] 2024-09-16T13:30:46.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 77afb3dde41e5946b77c75f36e73d56d != \7\7\a\f\b\3\d\d\e\4\1\e\5\9\4\6\b\7\7\c\7\5\f\3\6\e\7\3\d\5\6\d ]] 2024-09-16T13:30:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=630b18590e7ee23f9005853cf442bd1a 2024-09-16T13:30:46.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd1 2024-09-16T13:30:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=40326a84d8a7f5dbff4c2ae784e7b6ff 2024-09-16T13:30:46.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-09-16T13:30:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 826d37228277ba040cfa676eb8087955 != \8\2\6\d\3\7\2\2\8\2\7\7\b\a\0\4\0\c\f\a\6\7\6\e\b\8\0\8\7\9\5\5 ]] 2024-09-16T13:30:46.401 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:30:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 116925 2024-09-16T13:30:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:30:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 116927 2024-09-16T13:30:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:30:46.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 116929 2024-09-16T13:30:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 5df1dfb7a0a5f49cc7d2d0dfed001d5d != \5\d\f\1\d\f\b\7\a\0\a\5\f\4\9\c\c\7\d\2\d\0\d\f\e\d\0\0\1\d\5\d ]] 2024-09-16T13:30:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:30:46.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 116930 2024-09-16T13:30:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 630b18590e7ee23f9005853cf442bd1a != \6\3\0\b\1\8\5\9\0\e\7\e\e\2\3\f\9\0\0\5\8\5\3\c\f\4\4\2\b\d\1\a ]] 2024-09-16T13:30:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:30:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 116932 2024-09-16T13:30:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:30:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 116934 2024-09-16T13:30:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:30:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 116936 2024-09-16T13:30:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 40326a84d8a7f5dbff4c2ae784e7b6ff != \4\0\3\2\6\a\8\4\d\8\a\7\f\5\d\b\f\f\4\c\2\a\e\7\8\4\e\7\b\6\f\f ]] 2024-09-16T13:30:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:30:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 116938 2024-09-16T13:30:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:30:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 116939 2024-09-16T13:30:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T13:30:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:30:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary1 2024-09-16T13:30:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-16T13:30:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:30:46.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:30:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:30:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:30:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:30:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:30:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:30:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:30:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-16T13:30:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:30:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:30:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:30:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:30:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:30:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:30:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:30:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:30:48.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T13:30:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:30:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: ab41f43b-2d91-4d86-a967-20ff72553573 2024-09-16T13:30:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:30:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70823936.0,"last_snapshot_bytes":70823936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"} 2024-09-16T13:30:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:30:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:30:23 2024-09-16T13:30:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:30:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:30:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:30:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:30:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:30:43' 2024-09-16T13:30:48.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:30:48.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:30:48.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:30:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70823936.0,"last_snapshot_bytes":70823936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"} 2024-09-16T13:30:48.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:30:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:30:48.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:30:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:30:48.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:30:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:30:48.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:30:48.422 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:30:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:30:23' 2024-09-16T13:30:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-16 13:30:23' 2024-09-16T13:30:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:30:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:30:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:30:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:30:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:30:23' 2024-09-16T13:30:48.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:30:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:30:48.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:30:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ab41f43b-2d91-4d86-a967-20ff72553573up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70823936.0,"last_snapshot_bytes":70823936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:43image-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:43image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134217728.0,"last_snapshot_bytes":134217728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:43image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70213632.0,"last_snapshot_bytes":70213632,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:43image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133693440.0,"last_snapshot_bytes":133693440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:43image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63344640.0,"last_snapshot_bytes":63344640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:43image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70127616.0,"last_snapshot_bytes":70127616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:43image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63877120.0,"last_snapshot_bytes":63877120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:43image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70180864.0,"last_snapshot_bytes":70180864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:43image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70221824.0,"last_snapshot_bytes":70221824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:43' 2024-09-16T13:30:48.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:30:48.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T13:30:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:30:23' 2024-09-16T13:30:48.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T13:30:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:30:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:30:48.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:30:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:30:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:30:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:30:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-16T13:30:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:30:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:30:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:30:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-16T13:30:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-16T13:30:50.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:30:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-16T13:30:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:30:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:30:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:30:50.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:30:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:30:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary1 2024-09-16T13:30:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:30:50.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary1 2024-09-16T13:30:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:30:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary2 2024-09-16T13:30:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-16T13:30:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:30:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:30:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:30:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:30:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:30:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:30:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:30:50.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:30:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-16T13:30:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:30:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:30:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:30:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:30:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:30:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:30:51.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:30:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:30:51.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T13:30:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:30:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 8dbcffb5-dab3-446c-89d2-a6bb77d01c73 2024-09-16T13:30:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:30:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134217728.0,"last_snapshot_bytes":134217728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"} 2024-09-16T13:30:51.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:30:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:30:23 2024-09-16T13:30:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:30:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:30:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:30:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:30:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:30:48' 2024-09-16T13:30:51.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:30:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:30:51.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:30:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134217728.0,"last_snapshot_bytes":134217728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"} 2024-09-16T13:30:51.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:30:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:30:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:30:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:30:51.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:30:51.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:30:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:30:51.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:30:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:30:23' 2024-09-16T13:30:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-16 13:30:23' 2024-09-16T13:30:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:30:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:30:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:30:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:30:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:30:23' 2024-09-16T13:30:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:30:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:30:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:30:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK942544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:48image-primary28dbcffb5-dab3-446c-89d2-a6bb77d01c73up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":134217728.0,"last_snapshot_bytes":134217728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:48image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70213632.0,"last_snapshot_bytes":70213632,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:48image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133693440.0,"last_snapshot_bytes":133693440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:48image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63344640.0,"last_snapshot_bytes":63344640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:48image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70127616.0,"last_snapshot_bytes":70127616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:48image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63877120.0,"last_snapshot_bytes":63877120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:48image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70180864.0,"last_snapshot_bytes":70180864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:48image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70221824.0,"last_snapshot_bytes":70221824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:48' 2024-09-16T13:30:52.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:30:52.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T13:30:52.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:30:23' 2024-09-16T13:30:52.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T13:30:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:30:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:30:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:30:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:30:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:30:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:30:52.177 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-16T13:30:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:30:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:30:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:30:52.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-16T13:30:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-16T13:30:52.544 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:30:52.527+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T13:30:52.544 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:30:52.531+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T13:30:52.546 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:30:52.535+0000 7f61b1855700 -1 reset not still connected to 0x557b3fbad790 2024-09-16T13:30:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:30:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-16T13:30:53.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:30:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:30:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:30:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:30:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:30:53.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary2 2024-09-16T13:30:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:30:53.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary2 2024-09-16T13:30:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:30:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary3 2024-09-16T13:30:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-16T13:30:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:30:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:30:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:30:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:30:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:30:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:30:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:30:53.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:30:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-16T13:30:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:30:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:30:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:30:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:30:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:30:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:30:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:30:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:30:54.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T13:30:54.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:30:54.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: da5b1f11-8a62-48f0-8749-2008dd7a75cc 2024-09-16T13:30:54.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:30:54.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70213632.0,"last_snapshot_bytes":70213632,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"} 2024-09-16T13:30:54.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:30:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:30:53 2024-09-16T13:30:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:30:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:30:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:30:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:30:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:30:53' 2024-09-16T13:30:54.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:30:54.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:30:54.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:30:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70213632.0,"last_snapshot_bytes":70213632,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"} 2024-09-16T13:30:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:30:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:30:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:30:54.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:30:54.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:30:54.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:30:54.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:30:54.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:30:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:30:53' 2024-09-16T13:30:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-16 13:30:53' 2024-09-16T13:30:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:30:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:30:54.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:30:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:30:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:30:53' 2024-09-16T13:30:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:30:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:30:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:30:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK842544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:53image-primary3da5b1f11-8a62-48f0-8749-2008dd7a75ccup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70213632.0,"last_snapshot_bytes":70213632,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:53image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133693440.0,"last_snapshot_bytes":133693440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:53image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63344640.0,"last_snapshot_bytes":63344640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:53image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70127616.0,"last_snapshot_bytes":70127616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:53image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63877120.0,"last_snapshot_bytes":63877120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:53image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70180864.0,"last_snapshot_bytes":70180864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:53image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70221824.0,"last_snapshot_bytes":70221824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:53' 2024-09-16T13:30:55.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:30:55.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T13:30:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:30:53' 2024-09-16T13:30:55.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T13:30:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:30:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:30:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:30:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:30:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:30:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:30:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-16T13:30:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:30:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:30:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:30:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-16T13:30:55.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-16T13:30:56.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:30:56.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-16T13:30:56.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:30:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:30:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:30:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:30:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:30:56.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary3 2024-09-16T13:30:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:30:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary3 2024-09-16T13:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary4 2024-09-16T13:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-16T13:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:30:56.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:30:56.934 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:30:56.919+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174d00 2024-09-16T13:30:56.937 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:30:56.927+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T13:30:56.937 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:30:56.927+0000 7ff4f6946700 -1 reset not still connected to 0x557563f56270 2024-09-16T13:30:56.938 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:30:56.927+0000 7ff4f6946700 -1 reset not still connected to 0x557563f568f0 2024-09-16T13:30:56.938 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:30:56.927+0000 7ff4f6946700 -1 reset not still connected to 0x557563f57ba0 2024-09-16T13:30:56.938 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:30:56.927+0000 7ff4f6946700 -1 reset not still connected to 0x557563f57ee0 2024-09-16T13:30:56.938 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:30:56.927+0000 7ff4f6946700 -1 reset not still connected to 0x557567347e10 2024-09-16T13:30:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-16T13:30:57.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:30:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:30:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:30:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:30:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:30:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:30:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:30:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:30:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T13:30:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:30:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a2ea9fe0-ce1d-400a-8b01-1f096f691b5c 2024-09-16T13:30:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:30:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133693440.0,"last_snapshot_bytes":133693440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"} 2024-09-16T13:30:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:30:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:30:53 2024-09-16T13:30:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:30:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:30:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:30:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:30:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:30:53' 2024-09-16T13:30:57.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:30:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:30:57.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:30:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133693440.0,"last_snapshot_bytes":133693440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"} 2024-09-16T13:30:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:30:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:30:57.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:30:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:30:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:30:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:30:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:30:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:30:57.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:30:53' 2024-09-16T13:30:57.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-16 13:30:53' 2024-09-16T13:30:57.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:30:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:30:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:30:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:30:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:30:53' 2024-09-16T13:30:57.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:30:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:30:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:30:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK742544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:53image-primary4a2ea9fe0-ce1d-400a-8b01-1f096f691b5cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":133693440.0,"last_snapshot_bytes":133693440,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:53image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63344640.0,"last_snapshot_bytes":63344640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:53image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70127616.0,"last_snapshot_bytes":70127616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:53image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63877120.0,"last_snapshot_bytes":63877120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:53image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70180864.0,"last_snapshot_bytes":70180864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:53image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70221824.0,"last_snapshot_bytes":70221824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:53' 2024-09-16T13:30:57.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:30:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T13:30:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:30:53' 2024-09-16T13:30:57.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T13:30:58.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:30:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:30:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:30:58.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:30:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:30:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:30:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-16T13:30:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:30:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:30:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:30:58.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-16T13:30:58.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-16T13:31:00.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:31:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-16T13:31:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:31:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:31:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:31:00.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary4 2024-09-16T13:31:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:31:00.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary4 2024-09-16T13:31:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:31:00.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary5 2024-09-16T13:31:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-16T13:31:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:31:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:31:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:31:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:31:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:31:00.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:31:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-16T13:31:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:31:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:31:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:31:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:31:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:31:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:31:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T13:31:01.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:31:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b48e7af7-8636-448d-b1d6-46b0539ab6e9 2024-09-16T13:31:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:31:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63344640.0,"last_snapshot_bytes":63344640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"} 2024-09-16T13:31:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:31:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:30:53 2024-09-16T13:31:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:31:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:31:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:31:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:31:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:30:58' 2024-09-16T13:31:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:31:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:31:01.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:31:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63344640.0,"last_snapshot_bytes":63344640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"} 2024-09-16T13:31:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:31:01.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:31:01.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:31:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:31:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:31:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:31:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:31:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:31:01.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:30:53' 2024-09-16T13:31:01.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-16 13:30:53' 2024-09-16T13:31:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:31:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:31:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:30:53' 2024-09-16T13:31:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:31:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:31:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:31:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK642544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:58image-primary5b48e7af7-8636-448d-b1d6-46b0539ab6e9up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63344640.0,"last_snapshot_bytes":63344640,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:58image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70127616.0,"last_snapshot_bytes":70127616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:58image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63877120.0,"last_snapshot_bytes":63877120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:58image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70180864.0,"last_snapshot_bytes":70180864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:58image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70221824.0,"last_snapshot_bytes":70221824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:30:58' 2024-09-16T13:31:02.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:31:02.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T13:31:02.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:30:53' 2024-09-16T13:31:02.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T13:31:02.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:31:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:31:02.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:31:02.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:31:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:31:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:31:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-16T13:31:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:31:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:31:02.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-16T13:31:02.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-16T13:31:02.549 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:02.539+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174d00 2024-09-16T13:31:02.549 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:02.539+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T13:31:02.549 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:02.539+0000 7ff4f6946700 -1 reset not still connected to 0x557561950680 2024-09-16T13:31:02.552 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:02.543+0000 7ff4f6946700 -1 reset not still connected to 0x557563f56270 2024-09-16T13:31:02.552 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:02.543+0000 7ff4f6946700 -1 reset not still connected to 0x557563f568f0 2024-09-16T13:31:02.552 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:02.543+0000 7ff4f6946700 -1 reset not still connected to 0x557563f57ba0 2024-09-16T13:31:02.552 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:02.543+0000 7ff4f6946700 -1 reset not still connected to 0x557563f57ee0 2024-09-16T13:31:04.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:31:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-16T13:31:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:31:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:31:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:31:04.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary5 2024-09-16T13:31:04.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:31:04.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary5 2024-09-16T13:31:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:31:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary6 2024-09-16T13:31:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-16T13:31:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:31:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:31:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:31:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:31:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:31:04.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:31:05.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-16T13:31:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:31:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:31:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:31:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:31:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:31:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:31:05.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T13:31:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:31:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 64ea0105-23d9-444f-a0cb-021f5706b8b1 2024-09-16T13:31:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:31:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70127616.0,"last_snapshot_bytes":70127616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"} 2024-09-16T13:31:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:31:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:30:53 2024-09-16T13:31:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:31:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:31:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:31:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:31:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:31:03' 2024-09-16T13:31:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:31:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:31:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:31:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70127616.0,"last_snapshot_bytes":70127616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"} 2024-09-16T13:31:05.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:31:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:31:05.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:31:05.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:31:05.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:31:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:31:05.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:31:05.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:31:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:30:53' 2024-09-16T13:31:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-16 13:30:53' 2024-09-16T13:31:05.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:31:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:31:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:30:53' 2024-09-16T13:31:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:31:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:31:05.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:31:05.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK542544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:31:03image-primary664ea0105-23d9-444f-a0cb-021f5706b8b1up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70127616.0,"last_snapshot_bytes":70127616,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:31:03image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63877120.0,"last_snapshot_bytes":63877120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:31:03image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70180864.0,"last_snapshot_bytes":70180864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:31:03image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70221824.0,"last_snapshot_bytes":70221824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:31:03' 2024-09-16T13:31:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:31:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T13:31:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:30:53' 2024-09-16T13:31:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T13:31:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:31:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:31:06.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:31:06.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:31:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:31:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:31:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-16T13:31:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:31:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:31:06.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-16T13:31:06.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-16T13:31:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:31:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-16T13:31:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:07.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:31:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:31:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:31:07.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary6 2024-09-16T13:31:07.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:31:07.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary6 2024-09-16T13:31:07.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:31:07.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary7 2024-09-16T13:31:07.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-16T13:31:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:31:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:31:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:31:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:31:07.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:31:07.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:31:08.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-16T13:31:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:31:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:31:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:31:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:31:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:31:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:31:08.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T13:31:08.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:31:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 7c2f6603-8476-43b9-946c-db51ec3d98a8 2024-09-16T13:31:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:31:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63877120.0,"last_snapshot_bytes":63877120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"} 2024-09-16T13:31:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:31:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:30:53 2024-09-16T13:31:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:31:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:31:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:31:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:31:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:31:08' 2024-09-16T13:31:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:31:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:31:08.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:31:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63877120.0,"last_snapshot_bytes":63877120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"} 2024-09-16T13:31:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:31:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:31:08.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:31:08.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:31:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:31:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:31:08.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:31:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:31:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:30:53' 2024-09-16T13:31:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-16 13:30:53' 2024-09-16T13:31:08.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:31:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:31:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:30:53' 2024-09-16T13:31:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:31:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:31:08.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:31:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK442544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:31:08image-primary77c2f6603-8476-43b9-946c-db51ec3d98a8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":63877120.0,"last_snapshot_bytes":63877120,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:31:08image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70180864.0,"last_snapshot_bytes":70180864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:31:08image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70221824.0,"last_snapshot_bytes":70221824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:31:08' 2024-09-16T13:31:09.107 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:31:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T13:31:09.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:30:53' 2024-09-16T13:31:09.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T13:31:09.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:31:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:31:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:31:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:31:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:31:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:31:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-16T13:31:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:31:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:31:09.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-16T13:31:09.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-16T13:31:10.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:31:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-16T13:31:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:31:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:31:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:31:10.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary7 2024-09-16T13:31:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:31:10.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary7 2024-09-16T13:31:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:31:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary8 2024-09-16T13:31:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-16T13:31:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:31:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:31:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:31:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:31:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:31:10.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:31:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-16T13:31:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:11.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:31:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:31:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:31:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:31:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:31:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:31:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T13:31:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:31:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 0be91fa0-3638-4a36-88bc-96f6a810458c 2024-09-16T13:31:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:31:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70180864.0,"last_snapshot_bytes":70180864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"} 2024-09-16T13:31:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:31:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:30:53 2024-09-16T13:31:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:31:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:31:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:31:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:31:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:31:08' 2024-09-16T13:31:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:31:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:31:11.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:31:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70180864.0,"last_snapshot_bytes":70180864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"} 2024-09-16T13:31:11.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:31:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:31:11.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:31:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:31:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:31:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:31:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:31:11.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:31:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:30:53' 2024-09-16T13:31:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-16 13:30:53' 2024-09-16T13:31:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:31:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:31:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:30:53' 2024-09-16T13:31:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:31:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:31:11.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:31:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK342544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:31:08image-primary80be91fa0-3638-4a36-88bc-96f6a810458cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70180864.0,"last_snapshot_bytes":70180864,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:31:08image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70221824.0,"last_snapshot_bytes":70221824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:31:08' 2024-09-16T13:31:12.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:31:12.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T13:31:12.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:30:53' 2024-09-16T13:31:12.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T13:31:12.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:31:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:31:12.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:31:12.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:31:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:31:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:31:12.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-16T13:31:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:31:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:31:12.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-16T13:31:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-16T13:31:13.353 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:13.339+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174d00 2024-09-16T13:31:13.353 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:13.339+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T13:31:13.353 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:13.339+0000 7ff4f6946700 -1 reset not still connected to 0x55756ccac820 2024-09-16T13:31:13.738 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:13.727+0000 7ff4f6946700 -1 reset not still connected to 0x55756ccac820 2024-09-16T13:31:13.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:31:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-16T13:31:13.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:31:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:31:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:31:13.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary8 2024-09-16T13:31:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:31:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary8 2024-09-16T13:31:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:31:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary9 2024-09-16T13:31:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-16T13:31:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:31:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:31:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:31:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:31:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:31:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:31:14.290 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:31:14.279+0000 7f61b1855700 -1 reset not still connected to 0x557b229664e0 2024-09-16T13:31:14.291 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:31:14.279+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T13:31:14.291 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:31:14.279+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T13:31:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-16T13:31:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:31:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:31:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:31:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:31:14.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:31:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:31:14.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T13:31:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:31:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 18f8853c-a255-4064-a250-a3c514f2d73d 2024-09-16T13:31:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:31:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70221824.0,"last_snapshot_bytes":70221824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"} 2024-09-16T13:31:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:31:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:30:53 2024-09-16T13:31:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:31:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:31:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:31:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:31:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:31:13' 2024-09-16T13:31:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:31:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:31:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:31:14.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70221824.0,"last_snapshot_bytes":70221824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"} 2024-09-16T13:31:14.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:31:14.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:31:14.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:31:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:31:14.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:31:14.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:31:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:31:14.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:31:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:30:53' 2024-09-16T13:31:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-16 13:30:53' 2024-09-16T13:31:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:31:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:31:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:30:53' 2024-09-16T13:31:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:31:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:31:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:31:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK242544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:31:13image-primary918f8853c-a255-4064-a250-a3c514f2d73dup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70221824.0,"last_snapshot_bytes":70221824,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:31:13' 2024-09-16T13:31:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:31:15.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T13:31:15.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:30:53' 2024-09-16T13:31:15.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T13:31:15.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:31:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:31:15.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:31:15.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:31:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:31:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:31:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-16T13:31:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:31:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:31:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-16T13:31:15.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-16T13:31:16.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:31:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-16T13:31:16.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:31:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:31:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:31:16.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary9 2024-09-16T13:31:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:31:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary9 2024-09-16T13:31:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:31:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary10 2024-09-16T13:31:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-16T13:31:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:31:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:31:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:31:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:31:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:31:16.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:31:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-16T13:31:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:31:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:31:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:31:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:31:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:31:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:31:17.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T13:31:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:31:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: c42e0dcc-72ae-43da-866a-52ef5e032769 2024-09-16T13:31:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:31:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"} 2024-09-16T13:31:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:31:17.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:30:53 2024-09-16T13:31:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:31:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:31:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:31:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:31:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:31:13' 2024-09-16T13:31:17.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:31:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:31:17.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:31:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"} 2024-09-16T13:31:17.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:31:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:31:17.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:31:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:31:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:31:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:31:17.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:31:17.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:31:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:30:53' 2024-09-16T13:31:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-16 13:30:53' 2024-09-16T13:31:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:31:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:31:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:30:53' 2024-09-16T13:31:17.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:31:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:31:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:31:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK142544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary10c42e0dcc-72ae-43da-866a-52ef5e032769up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70197248.0,"last_snapshot_bytes":70197248,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493347,"remote_snapshot_timestamp":1726493347,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:30:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:31:13' 2024-09-16T13:31:17.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:31:17.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T13:31:17.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:30:53' 2024-09-16T13:31:17.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T13:31:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:31:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:31:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:31:17.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:31:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:31:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:31:17.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-16T13:31:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:31:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:31:17.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-16T13:31:18.059 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-16T13:31:18.359 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:31:18.347+0000 7f61b1855700 -1 reset not still connected to 0x557b41ca72b0 2024-09-16T13:31:18.810 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:31:18.799+0000 7f61b1855700 -1 reset not still connected to 0x557b1f47fd40 2024-09-16T13:31:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:31:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-16T13:31:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:31:19.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:31:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:31:19.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary10 2024-09-16T13:31:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:31:19.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary10 2024-09-16T13:31:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..10} 2024-09-16T13:31:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS=() 2024-09-16T13:31:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS=() 2024-09-16T13:31:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS=() 2024-09-16T13:31:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS=() 2024-09-16T13:31:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ RET=0 2024-09-16T13:31:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:31:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary1 2024-09-16T13:31:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary1 2024-09-16T13:31:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-16T13:31:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:31:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:31:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-16T13:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-16T13:31:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:31:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-16T13:31:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:31:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:31:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-16T13:31:20.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:31:20.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-16T13:31:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary1': 2024-09-16T13:31:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:31:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:31:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:31:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 359549e667f 2024-09-16T13:31:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.359549e667f 2024-09-16T13:31:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:31:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:31:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:31:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:31:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:31:19 2024 2024-09-16T13:31:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:31:19 2024 2024-09-16T13:31:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:31:19 2024 2024-09-16T13:31:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:31:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:31:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: a186c2d5-ec70-413a-a351-4e491091d673 2024-09-16T13:31:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:31:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:31:20.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary1 2024-09-16T13:31:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd0 2024-09-16T13:31:21.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:31:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd0 2024-09-16T13:31:21.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:31:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:31:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:31:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: bc0e6a1b-08ab-4bc1-b10b-78deab4251ff 2024-09-16T13:31:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:31:21.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:31:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:31:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:31:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:31:21.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:31:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:31:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:31:21.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary1 2024-09-16T13:31:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd0 test-primary1 2024-09-16T13:31:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:31:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary1 2024-09-16T13:31:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:31:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-16T13:31:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:31:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:31:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:31:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:21.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary1 2024-09-16T13:31:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:31:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary2 2024-09-16T13:31:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary2 2024-09-16T13:31:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:31:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-16T13:31:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:31:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:21.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:31:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary1 2024-09-16T13:31:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:31:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-16T13:31:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:31:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-16T13:31:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:31:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:31:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:31:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:31:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-16T13:31:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:31:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-16T13:31:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:31:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:31:21.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-16T13:31:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:31:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-16T13:31:22.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5282 2024-09-16T13:31:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:31:22.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:22.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-16T13:31:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary2': 2024-09-16T13:31:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:31:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:31:23.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:31:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 35a47267c201 2024-09-16T13:31:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.35a47267c201 2024-09-16T13:31:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:31:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:31:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:31:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:31:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:31:21 2024 2024-09-16T13:31:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:31:21 2024 2024-09-16T13:31:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:31:21 2024 2024-09-16T13:31:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:31:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:31:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: b923f668-8fc9-4de6-9396-9c071c8843c5 2024-09-16T13:31:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:31:23.051 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:31:23.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary2 2024-09-16T13:31:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd1 2024-09-16T13:31:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:31:23.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd1 2024-09-16T13:31:23.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:31:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:31:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:31:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 8b8a5c81-303d-44a1-81ea-a16964b36ce4 2024-09-16T13:31:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:31:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:31:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:31:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:31:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:31:23.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:31:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:31:23.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:31:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary2 2024-09-16T13:31:23.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd1 test-primary2 2024-09-16T13:31:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:31:23.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary2 2024-09-16T13:31:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:31:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:31:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-16T13:31:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:31:23.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary3 2024-09-16T13:31:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary3 2024-09-16T13:31:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-16T13:31:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:31:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:31:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary2 2024-09-16T13:31:23.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:31:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:31:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:31:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-16T13:31:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:31:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary2 2024-09-16T13:31:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:31:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-16T13:31:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:31:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:31:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:31:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:31:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-16T13:31:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:31:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-16T13:31:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:31:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:31:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-16T13:31:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:31:24.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-16T13:31:24.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5284 2024-09-16T13:31:24.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:31:24.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-16T13:31:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary3': 2024-09-16T13:31:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:31:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:31:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:31:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 35b41c62853 2024-09-16T13:31:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.35b41c62853 2024-09-16T13:31:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:31:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:31:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:31:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:31:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:31:24 2024 2024-09-16T13:31:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:31:24 2024 2024-09-16T13:31:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:31:24 2024 2024-09-16T13:31:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:31:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:31:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: ff91f6f4-8d42-45ca-a2a7-8a79187744cb 2024-09-16T13:31:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:31:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:31:25.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary3 2024-09-16T13:31:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd2 2024-09-16T13:31:25.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:31:25.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd2 2024-09-16T13:31:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:31:25.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:31:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:31:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 503f7bf8-5440-4864-b80c-4822936e77b0 2024-09-16T13:31:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:31:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:31:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:31:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:31:25.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:31:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:31:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:31:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:31:25.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary3 2024-09-16T13:31:25.627 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd2 test-primary3 2024-09-16T13:31:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:31:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary3 2024-09-16T13:31:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:31:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-16T13:31:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:25.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:31:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:31:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:31:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:31:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary3 2024-09-16T13:31:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:31:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary4 2024-09-16T13:31:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary4 2024-09-16T13:31:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:31:25.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-16T13:31:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary3 2024-09-16T13:31:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-16T13:31:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:31:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:31:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:31:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-16T13:31:25.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:31:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:31:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:31:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:31:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:25.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-16T13:31:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:31:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-16T13:31:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:31:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:31:25.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-16T13:31:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:31:25.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-16T13:31:25.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5287 2024-09-16T13:31:25.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:31:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:31:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:31:26.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:31:26.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-16T13:31:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary4': 2024-09-16T13:31:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:31:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:31:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:31:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 35c4205b639b 2024-09-16T13:31:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.35c4205b639b 2024-09-16T13:31:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:31:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:31:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:31:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:31:27.025 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:31:25 2024 2024-09-16T13:31:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:31:25 2024 2024-09-16T13:31:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:31:25 2024 2024-09-16T13:31:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:31:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:31:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 01c77cad-81df-436b-bfc6-ce1e9fd16e59 2024-09-16T13:31:27.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:31:27.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:31:27.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary4 2024-09-16T13:31:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd3 2024-09-16T13:31:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:31:27.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd3 2024-09-16T13:31:27.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:31:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:31:27.287 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:31:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: ffde3aa4-f7ad-4de9-8736-e02b201ee417 2024-09-16T13:31:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:31:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:31:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:31:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:31:27.288 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:31:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:31:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:31:27.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:31:27.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary4 2024-09-16T13:31:27.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd3 test-primary4 2024-09-16T13:31:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:31:27.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary4 2024-09-16T13:31:27.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:31:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:31:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:31:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary4 2024-09-16T13:31:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary5 2024-09-16T13:31:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary5 2024-09-16T13:31:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-16T13:31:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary4 2024-09-16T13:31:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:27.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-16T13:31:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:31:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:31:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:31:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-16T13:31:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:27.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:31:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:31:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:31:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:31:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:27.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-16T13:31:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-16T13:31:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:31:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:27.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:31:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:31:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:31:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:31:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-16T13:31:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:31:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:31:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-16T13:31:27.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:31:27.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-16T13:31:27.907 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:27.895+0000 7ff4f6946700 -1 reset not still connected to 0x55753c1b1380 2024-09-16T13:31:27.909 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:27.899+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174340 2024-09-16T13:31:27.909 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:27.899+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1748f0 2024-09-16T13:31:27.909 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:27.899+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174d00 2024-09-16T13:31:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5290 2024-09-16T13:31:27.916 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:27.903+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T13:31:27.916 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:27.903+0000 7ff4f6946700 -1 reset not still connected to 0x55757a4b15f0 2024-09-16T13:31:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:27.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:31:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:31:27.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:31:27.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:27.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5293 2024-09-16T13:31:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:31:28.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-16T13:31:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary5': 2024-09-16T13:31:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:31:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:31:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:31:28.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 35d55973c4b9 2024-09-16T13:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.35d55973c4b9 2024-09-16T13:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:31:27 2024 2024-09-16T13:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:31:27 2024 2024-09-16T13:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:31:27 2024 2024-09-16T13:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: c3613aef-b1a7-426c-babb-f25deca9a330 2024-09-16T13:31:28.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:31:28.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:31:28.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary5 2024-09-16T13:31:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:31:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:31:28.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:31:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd4 2024-09-16T13:31:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:31:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd4 2024-09-16T13:31:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:31:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/26214401052672/2621440 done 2024-09-16T13:31:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:31:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 8e5fea18-c832-42c7-8516-24840c0a77d5 2024-09-16T13:31:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:31:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:31:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:31:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:31:29.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:31:29.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:31:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:31:29.514 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:31:29.515 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary5 2024-09-16T13:31:29.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd4 test-primary5 2024-09-16T13:31:29.615 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5294 2024-09-16T13:31:29.628 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:31:29.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary5 2024-09-16T13:31:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:31:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-16T13:31:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:31:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:31:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary6 2024-09-16T13:31:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary6 2024-09-16T13:31:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-16T13:31:29.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:31:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:31:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:31:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-16T13:31:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:31:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:31:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:31:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:31:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-16T13:31:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:31:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:29.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:31:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:31:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:31:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary5 2024-09-16T13:31:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary5 2024-09-16T13:31:29.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-16T13:31:29.741 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:31:29.731+0000 7f61b1855700 -1 reset not still connected to 0x557b229664e0 2024-09-16T13:31:29.741 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:31:29.731+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T13:31:29.741 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:31:29.731+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T13:31:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:31:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-16T13:31:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:29.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:31:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:31:29.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-16T13:31:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:31:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-16T13:31:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5298 2024-09-16T13:31:30.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5300 2024-09-16T13:31:30.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:31:30.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-16T13:31:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary6': 2024-09-16T13:31:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:31:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:31:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:31:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 35edda626409 2024-09-16T13:31:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.35edda626409 2024-09-16T13:31:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:31:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:31:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:31:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:31:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:31:29 2024 2024-09-16T13:31:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:31:29 2024 2024-09-16T13:31:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:31:29 2024 2024-09-16T13:31:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:31:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:31:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 081de7ab-65ba-457e-8c2d-6dd87fef7409 2024-09-16T13:31:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:31:30.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:31:30.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary6 2024-09-16T13:31:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:31:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:31:30.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:31:30.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:31:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:31:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:31:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd5 2024-09-16T13:31:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:31:31.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd5 2024-09-16T13:31:31.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:31:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:31:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:31:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: c0801704-f317-4aa3-bd6d-19f93a396ce6 2024-09-16T13:31:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:31:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:31:31.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:31:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:31:31.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:31:32.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:31:32.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:31:32.535 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:31:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary6 2024-09-16T13:31:32.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd5 test-primary6 2024-09-16T13:31:32.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5303 2024-09-16T13:31:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5302 2024-09-16T13:31:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:31:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary6 2024-09-16T13:31:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:31:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:32.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:31:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:31:32.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:31:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-16T13:31:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:31:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:31:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary7 2024-09-16T13:31:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary7 2024-09-16T13:31:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-16T13:31:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:31:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary6 2024-09-16T13:31:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:31:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:31:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-16T13:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:31:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary6 2024-09-16T13:31:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:31:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:31:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:31:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-16T13:31:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-16T13:31:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-16T13:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-16T13:31:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:31:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-16T13:31:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:31:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:31:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:31:33.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:31:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5310 2024-09-16T13:31:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:31:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:31:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:31:33.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:33.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-16T13:31:33.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary7': 2024-09-16T13:31:33.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:31:33.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:31:33.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:31:33.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 36041a941d2b 2024-09-16T13:31:33.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.36041a941d2b 2024-09-16T13:31:33.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:31:33.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:31:33.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:31:33.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:31:33.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:31:32 2024 2024-09-16T13:31:33.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:31:32 2024 2024-09-16T13:31:33.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:31:32 2024 2024-09-16T13:31:33.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:31:33.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:31:33.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 926c5e9b-1ee1-4c34-af9d-6cb33936c71f 2024-09-16T13:31:33.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:31:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:31:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary7 2024-09-16T13:31:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd6 2024-09-16T13:31:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:31:34.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd6 2024-09-16T13:31:34.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:31:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:31:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:31:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 9abd3c28-d92f-4093-b79f-175bcd3ff1e0 2024-09-16T13:31:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:31:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:31:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:31:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:31:34.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:31:34.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:31:34.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 Snapshot ID: 5309 2024-09-16T13:31:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T13:31:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:31:34.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary7 2024-09-16T13:31:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd6 test-primary7 2024-09-16T13:31:34.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:34.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:31:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary7 2024-09-16T13:31:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:31:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-16T13:31:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:31:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:35.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:31:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary7 2024-09-16T13:31:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:31:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:31:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary8 2024-09-16T13:31:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary8 2024-09-16T13:31:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary7 2024-09-16T13:31:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-16T13:31:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-16T13:31:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:31:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:31:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:31:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:31:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:31:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-16T13:31:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:35.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:31:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:31:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:31:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:31:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-16T13:31:35.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:31:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-16T13:31:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:31:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:31:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-16T13:31:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:31:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-16T13:31:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:31:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:31:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:31:35.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:31:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:31:35.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:31:35.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5313 2024-09-16T13:31:35.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5312 2024-09-16T13:31:35.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:31:35.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5317 2024-09-16T13:31:35.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-16T13:31:35.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary8': 2024-09-16T13:31:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:31:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:31:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:31:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 3617a8802b6f 2024-09-16T13:31:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.3617a8802b6f 2024-09-16T13:31:35.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:31:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:31:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:31:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:31:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:31:35 2024 2024-09-16T13:31:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:31:35 2024 2024-09-16T13:31:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:31:35 2024 2024-09-16T13:31:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:31:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:31:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 49499ffb-8600-4c6a-a77d-3131161f82c8 2024-09-16T13:31:35.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:31:35.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:31:35.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary8 2024-09-16T13:31:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd7 2024-09-16T13:31:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:31:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd7 2024-09-16T13:31:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:31:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:31:36.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:31:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 00af9ba6-b072-4f2a-a78b-a83291d927ea 2024-09-16T13:31:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:31:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:31:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:31:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:31:36.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:31:36.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:31:36.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:31:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:31:36.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary8 2024-09-16T13:31:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd7 test-primary8 2024-09-16T13:31:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:31:36.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:31:36.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:31:36.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:31:36.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary8 2024-09-16T13:31:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:31:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-16T13:31:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:31:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:31:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary9 2024-09-16T13:31:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary9 2024-09-16T13:31:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-16T13:31:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:31:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:31:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:31:36.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary8 2024-09-16T13:31:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:31:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:31:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:31:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-16T13:31:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:31:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:36.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:31:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:31:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:31:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:31:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-16T13:31:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary8 2024-09-16T13:31:36.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-16T13:31:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:31:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-16T13:31:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:31:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:31:36.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-16T13:31:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:31:36.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-16T13:31:37.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5320 2024-09-16T13:31:37.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5324 2024-09-16T13:31:37.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5319 2024-09-16T13:31:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:31:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:31:37.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:31:37.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:31:37.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-16T13:31:38.029 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:38.018+0000 7ff4f6946700 -1 reset not still connected to 0x55758f535380 2024-09-16T13:31:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary9': 2024-09-16T13:31:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:31:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:31:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:31:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 363186a3236f 2024-09-16T13:31:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.363186a3236f 2024-09-16T13:31:38.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:31:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:31:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:31:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:31:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:31:36 2024 2024-09-16T13:31:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:31:36 2024 2024-09-16T13:31:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:31:36 2024 2024-09-16T13:31:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:31:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:31:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 03a85e26-8103-4730-a53c-df66946ae4b5 2024-09-16T13:31:38.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:31:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:31:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary9 2024-09-16T13:31:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd8 2024-09-16T13:31:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:31:38.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd8 2024-09-16T13:31:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:31:38.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:31:38.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:31:38.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: f2cf5636-eeb7-4454-8ad9-eaadd811a238 2024-09-16T13:31:38.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:31:38.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:31:38.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:31:38.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:31:38.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:31:38.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): Snapshot ID: 5323 2024-09-16T13:31:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:31:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:31:38.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:31:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:31:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:31:38.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:31:38.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T13:31:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:31:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:31:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:31:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:31:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:31:38.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary9 2024-09-16T13:31:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd8 test-primary9 2024-09-16T13:31:38.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:31:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary9 2024-09-16T13:31:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:31:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-16T13:31:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:31:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:31:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:31:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:31:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:31:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary9 2024-09-16T13:31:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:31:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary10 2024-09-16T13:31:38.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary10 2024-09-16T13:31:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary9 2024-09-16T13:31:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-16T13:31:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-16T13:31:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:31:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:31:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:31:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:38.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-16T13:31:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:31:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:31:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:31:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:31:38.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:31:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-16T13:31:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:31:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-16T13:31:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:31:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:31:39.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-16T13:31:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:31:39.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-16T13:31:39.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5328 2024-09-16T13:31:39.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:39.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5334 2024-09-16T13:31:39.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:31:39.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:39.855 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:39.846+0000 7ff4f6946700 -1 reset not still connected to 0x55756db6cf70 2024-09-16T13:31:39.855 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:39.846+0000 7ff4f6946700 -1 reset not still connected to 0x55758f535380 2024-09-16T13:31:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-16T13:31:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary10': 2024-09-16T13:31:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:31:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:31:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:31:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 364defd1198b 2024-09-16T13:31:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.364defd1198b 2024-09-16T13:31:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:31:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:31:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:31:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:31:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:31:38 2024 2024-09-16T13:31:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:31:38 2024 2024-09-16T13:31:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:31:38 2024 2024-09-16T13:31:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:31:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:31:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 55780b8f-37f1-4c9d-99fe-a496489bb95e 2024-09-16T13:31:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:31:40.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:31:40.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary10 2024-09-16T13:31:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd9 2024-09-16T13:31:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:31:40.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd9 2024-09-16T13:31:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:31:40.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:31:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:31:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: af7361c9-dfd8-41dc-8cd2-f51480bd3d13 2024-09-16T13:31:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:31:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:31:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:31:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:31:40.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:31:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:31:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:31:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:31:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:31:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:31:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:31:40.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:31:40.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 Snapshot ID: 5330 2024-09-16T13:31:40.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:31:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:31:40.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:31:40.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:40.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5331 2024-09-16T13:31:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T13:31:40.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:31:40.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary10 2024-09-16T13:31:40.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:40.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5333 2024-09-16T13:31:40.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd9 test-primary10 2024-09-16T13:31:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:41.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:31:41.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary10 2024-09-16T13:31:41.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:31:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-16T13:31:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:31:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:31:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:31:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 120645 2024-09-16T13:31:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:31:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary10 2024-09-16T13:31:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:31:41.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:31:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary10 2024-09-16T13:31:41.103 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-16T13:31:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:31:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-16T13:31:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:31:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:31:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:31:41.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5343 2024-09-16T13:31:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:42.017 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:42.010+0000 7ff4f6946700 -1 reset not still connected to 0x55756afb4d00 2024-09-16T13:31:42.017 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:31:42.010+0000 7ff4f6946700 -1 reset not still connected to 0x55756db6cf70 2024-09-16T13:31:42.646 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:31:42.635+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e36aa000 2024-09-16T13:31:42.646 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:31:42.635+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e36ab1e0 2024-09-16T13:31:42.653 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:31:42.639+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e5d9d2b0 2024-09-16T13:31:42.653 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:31:42.639+0000 7f315ffe6700 -1 reset not still connected to 0x55c1ff90cc30 2024-09-16T13:31:42.653 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:31:42.639+0000 7f315ffe6700 -1 reset not still connected to 0x55c204d6d860 2024-09-16T13:31:42.653 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:31:42.639+0000 7f315ffe6700 -1 reset not still connected to 0x55c20a2b1ee0 2024-09-16T13:31:42.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5340 2024-09-16T13:31:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5342 2024-09-16T13:31:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5341 2024-09-16T13:31:42.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:31:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:31:42.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:31:42.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:42.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:42.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:31:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:31:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:31:42.936 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:31:42.926+0000 7f61b1855700 -1 reset not still connected to 0x557b1f47fd40 2024-09-16T13:31:42.940 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:31:42.930+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T13:31:42.940 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:31:42.930+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T13:31:43.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5346 2024-09-16T13:31:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:31:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:31:43.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:31:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:31:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:31:43.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:31:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:43.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:31:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:31:43.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:31:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5348 2024-09-16T13:31:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:44.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5350 2024-09-16T13:31:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:31:44.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:31:44.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:31:45.436 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:31:45.427+0000 7f74def64700 -1 reset not still connected to 0x55945a3ee410 2024-09-16T13:31:45.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5354 2024-09-16T13:31:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5355 2024-09-16T13:31:45.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:45.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:31:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:31:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:31:45.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5352 2024-09-16T13:31:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:31:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:31:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:31:45.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:31:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:31:45.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:31:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5359 2024-09-16T13:31:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:31:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:31:46.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:31:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:31:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:31:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:31:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5361 2024-09-16T13:31:47.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5362 2024-09-16T13:31:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:47.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:31:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:31:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:31:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5363 2024-09-16T13:31:47.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:48.621 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5367 2024-09-16T13:31:48.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:31:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:31:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:31:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:31:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:31:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:31:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:31:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:31:48.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:31:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:31:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:31:49.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:31:49.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5370 2024-09-16T13:31:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:49.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5369 2024-09-16T13:31:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5374 2024-09-16T13:31:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:31:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:50.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:31:50.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:31:50.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:50.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5375 2024-09-16T13:31:50.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5373 2024-09-16T13:31:50.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:31:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:31:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:31:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:50.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:31:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:31:50.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:31:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:31:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:31:51.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:31:51.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5377 2024-09-16T13:31:51.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:52.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5381 2024-09-16T13:31:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:31:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:31:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:31:52.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5383 2024-09-16T13:31:52.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5382 2024-09-16T13:31:52.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:52.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:31:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:31:52.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:31:53.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5384 2024-09-16T13:31:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:31:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:31:53.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:31:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:31:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:31:53.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:31:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:31:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:31:53.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:31:54.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5389 2024-09-16T13:31:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:31:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:54.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:54.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:31:54.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:31:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5390 2024-09-16T13:31:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:54.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:55.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:31:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:31:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:31:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:31:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:31:55.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:31:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:55.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:31:55.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:55.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:55.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:31:55.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:31:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5394 2024-09-16T13:31:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5393 2024-09-16T13:31:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5395 2024-09-16T13:31:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:55.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:31:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:31:56.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:31:56.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5399 2024-09-16T13:31:56.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:57.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:57.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:31:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:31:57.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:31:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5402 2024-09-16T13:31:57.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5401 2024-09-16T13:31:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5400 2024-09-16T13:31:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:31:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:31:57.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:31:57.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:57.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5406 2024-09-16T13:31:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:31:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:31:58.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:31:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:31:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:31:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:31:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:31:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:31:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:31:59.629 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5410 2024-09-16T13:31:59.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5408 2024-09-16T13:31:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:59.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:31:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:31:59.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:31:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:31:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:31:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:31:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:32:00.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5413 2024-09-16T13:32:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5414 2024-09-16T13:32:00.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:00.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5415 2024-09-16T13:32:00.685 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:32:00.679+0000 7f74def64700 -1 reset not still connected to 0x5594654491e0 2024-09-16T13:32:00.685 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:32:00.679+0000 7f74def64700 -1 reset not still connected to 0x559465f18d00 2024-09-16T13:32:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:32:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:32:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:32:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:32:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:32:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:32:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:32:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:00.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:32:00.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:32:01.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5419 2024-09-16T13:32:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:32:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:32:01.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:32:01.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5422 2024-09-16T13:32:02.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5423 2024-09-16T13:32:02.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5421 2024-09-16T13:32:02.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:32:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:32:02.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:32:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:02.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:32:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:32:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:32:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:03.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:32:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:32:03.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:32:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:32:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:32:03.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:32:03.658 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5425 2024-09-16T13:32:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:03.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:03.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:32:03.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:03.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:03.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:32:03.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:32:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:32:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:32:04.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:32:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5429 2024-09-16T13:32:04.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:04.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5428 2024-09-16T13:32:04.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:05.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:32:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:32:05.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:32:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:32:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:32:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:32:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:32:05.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:32:05.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:32:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5433 2024-09-16T13:32:05.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5435 2024-09-16T13:32:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5431 2024-09-16T13:32:05.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:06.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:32:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5437 2024-09-16T13:32:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:06.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:32:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:32:06.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:07.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5440 2024-09-16T13:32:07.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5441 2024-09-16T13:32:07.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:07.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:32:07.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:07.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:07.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:32:07.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:32:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:32:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:32:07.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:32:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5443 2024-09-16T13:32:07.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:08.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5446 2024-09-16T13:32:08.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:08.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:32:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:32:08.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:32:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:32:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:32:08.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:32:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:32:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:32:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:32:09.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5449 2024-09-16T13:32:09.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5448 2024-09-16T13:32:09.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:32:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:32:09.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:32:09.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:32:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:32:10.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:32:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:32:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:32:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:32:10.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5453 2024-09-16T13:32:10.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5455 2024-09-16T13:32:10.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5454 2024-09-16T13:32:10.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:10.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:10.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:32:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:32:10.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:32:11.656 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:32:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:32:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:32:11.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5457 2024-09-16T13:32:11.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:11.705 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:32:11.698+0000 7ff4f6946700 -1 reset not still connected to 0x55756afb4d00 2024-09-16T13:32:11.705 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:32:11.698+0000 7ff4f6946700 -1 reset not still connected to 0x55756db6cf70 2024-09-16T13:32:12.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5462 2024-09-16T13:32:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:32:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:12.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:32:12.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:32:12.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5463 2024-09-16T13:32:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:32:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:32:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:32:12.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:12.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5461 2024-09-16T13:32:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5464 2024-09-16T13:32:13.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:32:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:32:13.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:32:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:32:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:32:13.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:32:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:32:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:32:13.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:32:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:32:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:32:14.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:32:14.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5470 2024-09-16T13:32:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5468 2024-09-16T13:32:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:14.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:15.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5473 2024-09-16T13:32:15.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5474 2024-09-16T13:32:15.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5475 2024-09-16T13:32:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:15.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:32:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:32:15.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:32:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:32:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:32:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:32:15.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:15.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:15.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:32:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:32:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:32:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5477 2024-09-16T13:32:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:32:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:32:16.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:32:16.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:32:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:32:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:32:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:32:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:32:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:32:17.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5480 2024-09-16T13:32:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5482 2024-09-16T13:32:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5481 2024-09-16T13:32:17.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:18.625 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5487 2024-09-16T13:32:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:32:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:32:18.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:32:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:32:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:32:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:32:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:32:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:32:18.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:32:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:32:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:32:19.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:32:19.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5489 2024-09-16T13:32:19.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5490 2024-09-16T13:32:19.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:19.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:20.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5495 2024-09-16T13:32:20.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5493 2024-09-16T13:32:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:32:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:32:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:32:20.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5494 2024-09-16T13:32:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:32:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:32:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:32:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:32:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:32:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:32:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:20.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:20.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:32:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:32:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:32:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5496 2024-09-16T13:32:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5503 2024-09-16T13:32:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5502 2024-09-16T13:32:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:32:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:32:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:32:22.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:22.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:32:22.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:22.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:22.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:32:22.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:32:22.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5501 2024-09-16T13:32:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:22.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:22.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5504 2024-09-16T13:32:23.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:32:23.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:32:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:32:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:32:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:23.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:32:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:32:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:32:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:32:23.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:32:24.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:32:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:32:24.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:32:24.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5509 2024-09-16T13:32:24.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5510 2024-09-16T13:32:24.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:25.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5515 2024-09-16T13:32:25.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5514 2024-09-16T13:32:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:32:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:32:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:32:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:32:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:32:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:32:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:32:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:32:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:32:25.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5513 2024-09-16T13:32:25.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:32:26.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:32:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:32:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5519 2024-09-16T13:32:26.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:26.858 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:32:26.850+0000 7ff4f6946700 -1 reset not still connected to 0x557566a99ba0 2024-09-16T13:32:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5522 2024-09-16T13:32:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5521 2024-09-16T13:32:27.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:27.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5523 2024-09-16T13:32:27.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:27.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:32:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:32:27.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:32:27.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:32:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:27.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:32:27.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:32:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5524 2024-09-16T13:32:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:32:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:32:28.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:32:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:32:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:32:28.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:32:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:32:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:32:28.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:32:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5530 2024-09-16T13:32:29.654 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5529 2024-09-16T13:32:29.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:29.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:32:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:32:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:32:29.876 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:32:29.870+0000 7ff4f6946700 -1 reset not still connected to 0x55755e643380 2024-09-16T13:32:29.884 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:32:29.874+0000 7ff4f6946700 -1 reset not still connected to 0x55756a7d8dd0 2024-09-16T13:32:29.884 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:32:29.874+0000 7ff4f6946700 -1 reset not still connected to 0x55756afb4d00 2024-09-16T13:32:29.884 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:32:29.874+0000 7ff4f6946700 -1 reset not still connected to 0x55756db6cf70 2024-09-16T13:32:29.884 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:32:29.878+0000 7ff4f6946700 -1 reset not still connected to 0x55758f535380 2024-09-16T13:32:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5534 2024-09-16T13:32:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5535 2024-09-16T13:32:30.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5533 2024-09-16T13:32:30.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:30.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:32:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:32:30.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:32:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:32:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:32:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:32:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:32:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:32:30.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:32:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:32:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:32:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:32:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5539 2024-09-16T13:32:31.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5542 2024-09-16T13:32:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:32:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:32:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:32:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:32:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:32:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5541 2024-09-16T13:32:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:32:32.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5543 2024-09-16T13:32:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:32.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5547 2024-09-16T13:32:33.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:32:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:32:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:32:33.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:33.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:32:33.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:33.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:33.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:32:33.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:32:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:32:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:32:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:32:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:32:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:32:34.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:32:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5550 2024-09-16T13:32:34.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5549 2024-09-16T13:32:34.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:34.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:35.669 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5555 2024-09-16T13:32:35.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:32:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:32:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:32:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:32:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:32:35.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:32:35.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5553 2024-09-16T13:32:35.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5552 2024-09-16T13:32:35.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:32:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:32:35.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:32:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:36.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:32:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:32:36.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:32:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5557 2024-09-16T13:32:36.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:37.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5561 2024-09-16T13:32:37.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5560 2024-09-16T13:32:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5563 2024-09-16T13:32:37.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:32:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:32:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:37.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:32:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:32:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:32:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:32:37.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:38.653 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5567 2024-09-16T13:32:38.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:32:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:32:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:32:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:32:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:32:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:32:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:32:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:32:38.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:32:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:32:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:32:39.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:32:39.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5569 2024-09-16T13:32:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:39.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5568 2024-09-16T13:32:39.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:32:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:32:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:32:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:32:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:40.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:32:40.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:32:40.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5574 2024-09-16T13:32:40.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5573 2024-09-16T13:32:40.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:32:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:32:40.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:32:40.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5575 2024-09-16T13:32:40.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:32:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:32:41.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:32:41.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5577 2024-09-16T13:32:41.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:42.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5581 2024-09-16T13:32:42.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5583 2024-09-16T13:32:42.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5580 2024-09-16T13:32:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:32:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:32:42.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:32:42.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:42.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:32:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:32:42.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:32:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5584 2024-09-16T13:32:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:32:43.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:32:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:32:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:32:43.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:32:43.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:32:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:32:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:32:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:32:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:32:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:32:44.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:32:44.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5590 2024-09-16T13:32:44.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5588 2024-09-16T13:32:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:44.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:44.785 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:32:44.778+0000 7ff4f6946700 -1 reset not still connected to 0x55756afb4d00 2024-09-16T13:32:44.785 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:32:44.778+0000 7ff4f6946700 -1 reset not still connected to 0x55756db6cf70 2024-09-16T13:32:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:45.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:32:45.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:45.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:45.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:32:45.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:32:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5593 2024-09-16T13:32:45.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5594 2024-09-16T13:32:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:32:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:32:45.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:32:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:32:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:32:45.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:32:45.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5595 2024-09-16T13:32:45.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:45.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:32:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:32:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:32:46.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5597 2024-09-16T13:32:46.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:32:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:32:47.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:32:47.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5600 2024-09-16T13:32:47.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:47.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:32:47.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:47.749 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:32:47.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:32:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5602 2024-09-16T13:32:47.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5601 2024-09-16T13:32:47.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:48.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5607 2024-09-16T13:32:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:32:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:32:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:32:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:48.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:32:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:32:48.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:32:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:32:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:32:48.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:32:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:32:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:32:49.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:32:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5609 2024-09-16T13:32:49.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5608 2024-09-16T13:32:49.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:49.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:32:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:32:50.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:32:50.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5613 2024-09-16T13:32:50.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5615 2024-09-16T13:32:50.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5614 2024-09-16T13:32:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:32:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:50.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:32:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:32:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:32:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:32:50.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:32:50.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:50.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:32:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:32:51.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:32:51.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5617 2024-09-16T13:32:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5622 2024-09-16T13:32:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:32:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:32:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:32:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:32:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:52.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:32:52.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:32:52.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5621 2024-09-16T13:32:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5620 2024-09-16T13:32:52.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:52.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:53.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5624 2024-09-16T13:32:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:32:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:53.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:32:53.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:32:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:32:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:32:53.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:32:53.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:53.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:32:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:32:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:32:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:32:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:32:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:32:54.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5629 2024-09-16T13:32:54.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5630 2024-09-16T13:32:54.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:54.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:55.596 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:32:55.587+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8bf12bad0 2024-09-16T13:32:55.596 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:32:55.587+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c299a4e0 2024-09-16T13:32:55.599 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:32:55.591+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c299aa90 2024-09-16T13:32:55.600 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:32:55.591+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c299bd40 2024-09-16T13:32:55.600 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:32:55.591+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c8c76ea0 2024-09-16T13:32:55.600 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:32:55.591+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8df23b110 2024-09-16T13:32:55.601 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:32:55.591+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8df6200d0 2024-09-16T13:32:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:32:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:32:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:32:55.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5635 2024-09-16T13:32:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5633 2024-09-16T13:32:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:32:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:55.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:32:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:32:55.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5634 2024-09-16T13:32:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:32:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:32:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:32:55.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:56.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5638 2024-09-16T13:32:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:32:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:32:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:32:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:57.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5642 2024-09-16T13:32:57.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5640 2024-09-16T13:32:57.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5641 2024-09-16T13:32:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:57.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:57.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:32:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:32:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:32:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:32:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:32:57.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:32:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5647 2024-09-16T13:32:58.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:58.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:58.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:32:58.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:58.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:58.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:32:58.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:32:58.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:32:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:32:58.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:32:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:32:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:32:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:32:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5649 2024-09-16T13:32:59.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5650 2024-09-16T13:32:59.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:32:59.694 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:32:59.687+0000 7f315ffe6700 -1 reset not still connected to 0x55c204d6d860 2024-09-16T13:32:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:32:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:32:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:32:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:32:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:32:59.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:33:00.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5653 2024-09-16T13:33:00.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:00.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5654 2024-09-16T13:33:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5655 2024-09-16T13:33:00.676 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:00.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:00.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:33:00.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:00.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:33:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:33:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:33:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:33:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:33:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:00.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:33:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:33:00.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:33:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:33:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:01.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:01.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:33:01.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:33:01.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5659 2024-09-16T13:33:01.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:33:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:33:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:33:02.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5662 2024-09-16T13:33:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:33:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:33:02.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:33:02.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5661 2024-09-16T13:33:02.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:02.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5663 2024-09-16T13:33:02.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:33:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:33:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:33:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:33:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:33:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:33:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5667 2024-09-16T13:33:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:33:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:03.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:33:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:33:03.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5669 2024-09-16T13:33:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:33:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:33:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:33:04.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:04.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5670 2024-09-16T13:33:04.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:05.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5674 2024-09-16T13:33:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:33:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:33:05.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:33:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:33:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:33:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:33:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:33:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:05.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:33:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:33:05.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5673 2024-09-16T13:33:05.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:05.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5675 2024-09-16T13:33:05.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:05.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:06.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5679 2024-09-16T13:33:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:33:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:33:06.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:33:06.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:33:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:33:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:33:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5682 2024-09-16T13:33:07.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5683 2024-09-16T13:33:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:33:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:07.734 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:33:07.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:33:07.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5681 2024-09-16T13:33:07.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:07.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:08.648 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5687 2024-09-16T13:33:08.662 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:08.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:33:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:33:08.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:33:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:33:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:33:08.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:33:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:08.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:33:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:33:08.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:33:08.758 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:33:08.751+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e5d9d2b0 2024-09-16T13:33:08.759 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:33:08.751+0000 7f315ffe6700 -1 reset not still connected to 0x55c204d6d860 2024-09-16T13:33:09.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5689 2024-09-16T13:33:09.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5690 2024-09-16T13:33:09.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:33:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:33:09.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:33:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:10.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5694 2024-09-16T13:33:10.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5695 2024-09-16T13:33:10.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5693 2024-09-16T13:33:10.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:10.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:33:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:33:10.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:33:10.746 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:33:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:33:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:33:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:33:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:10.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:33:10.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:33:11.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:11.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:33:11.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:11.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:11.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:33:11.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:33:11.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5699 2024-09-16T13:33:11.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:12.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5703 2024-09-16T13:33:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:33:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:33:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:33:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:33:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:33:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:33:12.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5701 2024-09-16T13:33:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5702 2024-09-16T13:33:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:12.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:12.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:13.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5704 2024-09-16T13:33:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:33:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:33:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:33:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:33:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:33:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:33:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:33:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:33:13.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:33:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:33:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:33:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:33:14.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5710 2024-09-16T13:33:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:14.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5709 2024-09-16T13:33:14.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:15.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5714 2024-09-16T13:33:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:33:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:15.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:33:15.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:33:15.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5713 2024-09-16T13:33:15.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5715 2024-09-16T13:33:15.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:33:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:15.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:33:15.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:33:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:33:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:33:15.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:33:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:15.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:16.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5719 2024-09-16T13:33:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:33:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:16.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:33:16.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:33:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:17.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:17.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:33:17.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:17.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:17.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:33:17.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:33:17.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5722 2024-09-16T13:33:17.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5720 2024-09-16T13:33:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:33:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:33:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:33:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5723 2024-09-16T13:33:17.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:17.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5727 2024-09-16T13:33:18.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:33:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:33:18.775 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:33:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:33:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:33:18.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:33:18.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:18.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:33:18.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:18.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:18.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:33:18.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:33:19.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5728 2024-09-16T13:33:19.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:19.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5730 2024-09-16T13:33:19.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:19.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:33:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:33:19.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:33:20.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5733 2024-09-16T13:33:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:33:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:33:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:33:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:33:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:33:20.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:33:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5734 2024-09-16T13:33:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5735 2024-09-16T13:33:20.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:33:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:33:20.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:33:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:20.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:33:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:33:21.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:33:21.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5739 2024-09-16T13:33:21.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:33:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:33:22.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:33:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:33:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:33:22.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:33:22.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5741 2024-09-16T13:33:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5742 2024-09-16T13:33:22.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5743 2024-09-16T13:33:22.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:22.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:22.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:23.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5744 2024-09-16T13:33:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:33:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:33:23.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:33:23.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:33:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:33:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:33:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:33:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:33:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:33:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:33:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:33:24.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:33:24.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5748 2024-09-16T13:33:24.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:24.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5749 2024-09-16T13:33:24.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:24.857 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:33:24.845+0000 7ff4f6946700 -1 reset not still connected to 0x55755e643380 2024-09-16T13:33:24.857 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:33:24.849+0000 7ff4f6946700 -1 reset not still connected to 0x557566297ee0 2024-09-16T13:33:24.859 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:33:24.849+0000 7ff4f6946700 -1 reset not still connected to 0x55756a7d8dd0 2024-09-16T13:33:24.861 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:33:24.853+0000 7ff4f6946700 -1 reset not still connected to 0x55756afb4d00 2024-09-16T13:33:24.861 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:33:24.853+0000 7ff4f6946700 -1 reset not still connected to 0x55756db6cf70 2024-09-16T13:33:24.861 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:33:24.853+0000 7ff4f6946700 -1 reset not still connected to 0x55758f535380 2024-09-16T13:33:25.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5755 2024-09-16T13:33:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5753 2024-09-16T13:33:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:33:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:33:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:33:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:33:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:33:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:33:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:33:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:33:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:33:25.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5754 2024-09-16T13:33:25.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:33:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:33:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:33:26.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5756 2024-09-16T13:33:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5762 2024-09-16T13:33:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:33:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:33:27.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:33:27.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5761 2024-09-16T13:33:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5763 2024-09-16T13:33:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:33:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:33:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:33:27.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:27.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:27.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:28.661 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5764 2024-09-16T13:33:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:33:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:33:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:28.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:33:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:33:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:33:28.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:33:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:33:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:33:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:33:29.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5768 2024-09-16T13:33:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:29.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5770 2024-09-16T13:33:29.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:33:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:33:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:33:30.652 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5774 2024-09-16T13:33:30.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5773 2024-09-16T13:33:30.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:30.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:30.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5775 2024-09-16T13:33:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:33:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:33:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:33:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:33:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:33:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:33:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:33:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:33:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:33:31.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5779 2024-09-16T13:33:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:33:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:33:31.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:33:31.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:31.737 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:33:31.729+0000 7ff4f6946700 -1 reset not still connected to 0x55758f535380 2024-09-16T13:33:32.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5783 2024-09-16T13:33:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:33:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:33:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:33:32.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5781 2024-09-16T13:33:32.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:32.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5782 2024-09-16T13:33:32.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:32.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:33:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:33:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:33:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:33:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:33:33.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:33:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:33:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:33:33.692 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:33:33.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5785 2024-09-16T13:33:33.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:33.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:33:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:33:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:33:33.720 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:33:33.713+0000 7f61b1855700 -1 reset not still connected to 0x557b57e0be10 2024-09-16T13:33:34.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:33:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:33:34.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:33:34.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5788 2024-09-16T13:33:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:34.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5790 2024-09-16T13:33:34.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5791 2024-09-16T13:33:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:33:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:33:35.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:33:35.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5795 2024-09-16T13:33:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:35.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:33:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:33:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:33:35.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:33:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:33:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:33:35.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5794 2024-09-16T13:33:35.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5796 2024-09-16T13:33:36.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:33:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:33:36.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:33:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5802 2024-09-16T13:33:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:33:37.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:33:37.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:33:37.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5801 2024-09-16T13:33:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5800 2024-09-16T13:33:37.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:37.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:33:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:33:37.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:33:38.675 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5807 2024-09-16T13:33:38.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:33:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:33:38.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:33:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:33:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:33:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:33:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:33:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:38.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:33:38.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:33:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:33:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:33:39.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:33:39.719 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5808 2024-09-16T13:33:39.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5810 2024-09-16T13:33:39.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:40.721 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5813 2024-09-16T13:33:40.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5814 2024-09-16T13:33:40.735 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:40.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5815 2024-09-16T13:33:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:33:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:33:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:33:40.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:33:40.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:33:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:33:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:33:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:33:40.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:33:40.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:41.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:41.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:33:41.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:41.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:41.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:33:41.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:33:41.740 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5817 2024-09-16T13:33:41.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:33:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:33:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:33:42.747 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5822 2024-09-16T13:33:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:33:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:33:42.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:33:42.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5823 2024-09-16T13:33:42.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:42.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5821 2024-09-16T13:33:42.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:43.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:33:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:33:43.742 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:33:43.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5826 2024-09-16T13:33:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:33:43.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:33:43.753 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:33:43.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:33:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:33:43.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:33:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:33:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:33:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:33:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5828 2024-09-16T13:33:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5829 2024-09-16T13:33:44.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:44.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:45.756 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5833 2024-09-16T13:33:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:33:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:33:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:33:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:33:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:33:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:33:45.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5835 2024-09-16T13:33:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5834 2024-09-16T13:33:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:33:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:45.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:33:45.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:33:45.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:33:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:33:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:33:46.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5838 2024-09-16T13:33:46.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:33:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:33:47.784 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:33:47.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5841 2024-09-16T13:33:47.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5840 2024-09-16T13:33:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:33:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:33:47.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:33:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5843 2024-09-16T13:33:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:47.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:47.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:48.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5847 2024-09-16T13:33:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:33:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:33:48.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:33:48.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:33:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:33:48.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:33:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:33:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:33:48.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:33:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5848 2024-09-16T13:33:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5850 2024-09-16T13:33:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:49.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:33:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:33:49.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:33:50.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5855 2024-09-16T13:33:50.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5854 2024-09-16T13:33:50.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5853 2024-09-16T13:33:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:50.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:33:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:33:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:33:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:33:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:33:50.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:33:50.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:33:50.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 120781 2024-09-16T13:33:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:33:51.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:33:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:33:51.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5859 2024-09-16T13:33:51.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:52.674 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5862 2024-09-16T13:33:52.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5861 2024-09-16T13:33:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:33:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 120915 2024-09-16T13:33:52.694 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:52.695 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:33:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:33:52.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:33:53.671 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5864 2024-09-16T13:33:53.685 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:33:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:33:53.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:33:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:33:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:33:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 121070 2024-09-16T13:33:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:53.704 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:33:53.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:33:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:33:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:33:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:33:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5867 2024-09-16T13:33:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:55.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5869 2024-09-16T13:33:55.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:33:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 121245 2024-09-16T13:33:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:33:55.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:33:55.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:33:55.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:55.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5870 2024-09-16T13:33:55.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:33:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:33:56.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:33:56.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5873 2024-09-16T13:33:56.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:57.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5875 2024-09-16T13:33:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:33:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:33:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:33:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:58.673 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5877 2024-09-16T13:33:58.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:33:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:33:58.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 121445 2024-09-16T13:33:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:33:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:33:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:33:59.705 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5879 2024-09-16T13:33:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:33:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:33:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:33:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:33:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:33:59.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:33:59.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:34:00.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5881 2024-09-16T13:34:00.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:34:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:34:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:34:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:34:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:34:00.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:34:00.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:34:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:34:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 121640 2024-09-16T13:34:01.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5883 2024-09-16T13:34:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:34:02.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5885 2024-09-16T13:34:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:34:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:34:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:34:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:34:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:34:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:34:02.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:34:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:34:03.720 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 121836 2024-09-16T13:34:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:34:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:34:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:34:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:34:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:34:04.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:34:04.743 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5887 2024-09-16T13:34:04.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:34:05.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:34:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 122091 2024-09-16T13:34:06.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5889 2024-09-16T13:34:06.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:34:07.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:34:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 122289 2024-09-16T13:34:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T13:34:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:34:09.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 120646 2024-09-16T13:36:21.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:36:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:36:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:36:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 120782 2024-09-16T13:36:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:36:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:36:24.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:36:24.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 120916 2024-09-16T13:36:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:36:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:36:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:36:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 121071 2024-09-16T13:36:27.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:36:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:36:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:36:27.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 121246 2024-09-16T13:36:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:36:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:36:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:36:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 121446 2024-09-16T13:36:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:36:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:36:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:36:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 121641 2024-09-16T13:36:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:36:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:36:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:36:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 121837 2024-09-16T13:36:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:36:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:36:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:36:36.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 122092 2024-09-16T13:36:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:36:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:36:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:36:39.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 122290 2024-09-16T13:36:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:36:41.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:36:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T13:36:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:36:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:36:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:36:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 1 2024-09-16T13:36:41.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:36:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:36:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 2 2024-09-16T13:36:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd0 2024-09-16T13:36:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary1 2024-09-16T13:36:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary1 2024-09-16T13:36:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:36:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:36:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary1 2024-09-16T13:36:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:36:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd1 2024-09-16T13:36:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary2 2024-09-16T13:36:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary2 2024-09-16T13:36:41.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:36:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 3 2024-09-16T13:36:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary2 2024-09-16T13:36:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:36:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd2 2024-09-16T13:36:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:36:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary3 2024-09-16T13:36:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary3 2024-09-16T13:36:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 4 2024-09-16T13:36:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:36:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary3 2024-09-16T13:36:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd3 2024-09-16T13:36:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:36:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary4 2024-09-16T13:36:41.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary4 2024-09-16T13:36:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:36:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:36:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary4 2024-09-16T13:36:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 5 2024-09-16T13:36:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd4 2024-09-16T13:36:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:36:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary5 2024-09-16T13:36:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary5 2024-09-16T13:36:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:36:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:36:41.246 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary5 2024-09-16T13:36:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:36:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 6 2024-09-16T13:36:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:36:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 7 2024-09-16T13:36:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd5 2024-09-16T13:36:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:36:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:36:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd6 2024-09-16T13:36:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary7 2024-09-16T13:36:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary7 2024-09-16T13:36:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:36:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 8 2024-09-16T13:36:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary7 2024-09-16T13:36:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary6 2024-09-16T13:36:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:36:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:36:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd7 2024-09-16T13:36:41.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary8 2024-09-16T13:36:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary8 2024-09-16T13:36:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:36:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary8 2024-09-16T13:36:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary6 2024-09-16T13:36:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:36:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:36:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary6 2024-09-16T13:36:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:36:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 128165 2024-09-16T13:36:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 10 2024-09-16T13:36:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 9 2024-09-16T13:36:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd9 2024-09-16T13:36:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary10 2024-09-16T13:36:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary10 2024-09-16T13:36:41.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:36:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary10 2024-09-16T13:36:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd8 2024-09-16T13:36:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary9 2024-09-16T13:36:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary9 2024-09-16T13:36:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:36:41.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary9 2024-09-16T13:36:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd8 2024-09-16T13:36:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:36:41.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:36:41.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd0 2024-09-16T13:36:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd3 2024-09-16T13:36:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:36:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd1 2024-09-16T13:36:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:36:41.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd5 2024-09-16T13:36:41.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:36:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:36:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd2 2024-09-16T13:36:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd6 2024-09-16T13:36:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:36:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd4 2024-09-16T13:36:41.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:36:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:36:41.800 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd9 2024-09-16T13:36:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd7 2024-09-16T13:36:41.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:37:37.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=8ba0351299f182d21361b09ccf60429b 2024-09-16T13:37:37.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary6 2024-09-16T13:37:38.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=1e185adf23d636df7c4627a009087a0d 2024-09-16T13:37:38.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary1 2024-09-16T13:37:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=9b234a348fa17bc0a853589d3a349ba8 2024-09-16T13:37:38.501 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary4 2024-09-16T13:37:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=3a56e2b07dcf980648d5a577d85cae12 2024-09-16T13:37:38.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary10 2024-09-16T13:37:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=e79e73bad3cccff91074d8413d336cf7 2024-09-16T13:37:38.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary9 2024-09-16T13:37:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=644922d124c736a1e2a7c64e41c88f87 2024-09-16T13:37:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary7 2024-09-16T13:37:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=f01371c5506c32e1bd3308fd65292ca5 2024-09-16T13:37:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary2 2024-09-16T13:37:39.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=1551a5ac1f610658954cb17610a962a7 2024-09-16T13:37:39.370 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary8 2024-09-16T13:37:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-16T13:37:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:37:39.528 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-16T13:37:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-16T13:37:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:37:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-16T13:37:39.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-16T13:37:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:39.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:37:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-16T13:37:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-16T13:37:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:39.716 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:37:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-16T13:37:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-16T13:37:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:39.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:37:39.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-16T13:37:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=5742100030069f1f3634fe7e07047941 2024-09-16T13:37:39.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary3 2024-09-16T13:37:39.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-16T13:37:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:37:39.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-16T13:37:40.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-16T13:37:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:37:40.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-16T13:37:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=476362081063b57a5f938436fbfd5330 2024-09-16T13:37:40.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary5 2024-09-16T13:37:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-16T13:37:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:37:40.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-16T13:37:40.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-16T13:37:40.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:40.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:40.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:37:40.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-16T13:37:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-16T13:37:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:37:40.499 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-16T13:37:41.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:37:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-16T13:37:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:37:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:37:41.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:37:41.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:37:41.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:37:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-16T13:37:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:37:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:37:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-16T13:37:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:37:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:41.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:37:41.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:37:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-16T13:37:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:37:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:41.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:37:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-16T13:37:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:37:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:41.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:41.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:37:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:37:42.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:37:42.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:37:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-16T13:37:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:37:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:42.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:37:42.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-16T13:37:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:37:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:42.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:42.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:37:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T13:37:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-16T13:37:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:37:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:37:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-16T13:37:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:37:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:37:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-16T13:37:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:37:42.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:37:42.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:37:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-16T13:37:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:37:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:42.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-16T13:37:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:37:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:42.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T13:37:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-16T13:37:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:37:42.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:42.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:37:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:42.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T13:37:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-16T13:37:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:37:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:42.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:42.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-16T13:37:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:37:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T13:37:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T13:37:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:37:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 081de7ab-65ba-457e-8c2d-6dd87fef7409 2024-09-16T13:37:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:37:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:37:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:38 2024-09-16T13:37:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:37:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:37:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":4265984.0,"last_snapshot_bytes":3801088,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493637,"remote_snapshot_timestamp":1726493637,"replay_state":"idle"} 2024-09-16T13:37:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:23' 2024-09-16T13:37:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:37:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:37:42.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:37:42.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 03a85e26-8103-4730-a53c-df66946ae4b5 2024-09-16T13:37:42.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:37:42.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:37:42.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:42.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:38 2024-09-16T13:37:42.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:42.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:37:42.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:42.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:37:42.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:41' 2024-09-16T13:37:42.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:37:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 01c77cad-81df-436b-bfc6-ce1e9fd16e59 2024-09-16T13:37:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:37:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:37:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:38 2024-09-16T13:37:42.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:37:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:37:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1708032.0,"last_snapshot_bytes":1761280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493631,"remote_snapshot_timestamp":1726493631,"replay_state":"idle"} 2024-09-16T13:37:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:23' 2024-09-16T13:37:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:37:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:42.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:37:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:37:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:42.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:37:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:37:42.396 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 55780b8f-37f1-4c9d-99fe-a496489bb95e 2024-09-16T13:37:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:37:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:37:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:38 2024-09-16T13:37:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:37:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:37:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2150400.0,"last_snapshot_bytes":2588672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493645,"remote_snapshot_timestamp":1726493645,"replay_state":"idle"} 2024-09-16T13:37:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:23' 2024-09-16T13:37:42.397 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:37:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a186c2d5-ec70-413a-a351-4e491091d673 2024-09-16T13:37:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:37:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:37:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:38 2024-09-16T13:37:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:37:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:37:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":3694592.0,"last_snapshot_bytes":7233536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493626,"remote_snapshot_timestamp":1726493626,"replay_state":"idle"} 2024-09-16T13:37:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:23' 2024-09-16T13:37:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:37:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:37:42.403 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:37:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:42.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:37:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-16T13:37:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:43.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:37:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:43.164 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T13:37:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-16T13:37:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:37:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:43.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T13:37:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-16T13:37:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:37:43.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:43.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:43.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:43.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:43.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:43.190 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T13:37:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-16T13:37:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:37:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T13:37:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-16T13:37:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:37:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T13:37:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:37:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: c3613aef-b1a7-426c-babb-f25deca9a330 2024-09-16T13:37:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:37:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:37:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:38 2024-09-16T13:37:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:37:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:37:43.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:42' 2024-09-16T13:37:43.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:37:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:43.252 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:37:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:37:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: ff91f6f4-8d42-45ca-a2a7-8a79187744cb 2024-09-16T13:37:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:37:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:37:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:38 2024-09-16T13:37:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:43.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:37:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:37:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:42' 2024-09-16T13:37:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:37:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:43.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:37:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:37:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 49499ffb-8600-4c6a-a77d-3131161f82c8 2024-09-16T13:37:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:37:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:37:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:38 2024-09-16T13:37:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:37:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:37:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:42' 2024-09-16T13:37:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:37:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:43.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:37:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:37:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 926c5e9b-1ee1-4c34-af9d-6cb33936c71f 2024-09-16T13:37:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:37:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:37:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:38 2024-09-16T13:37:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:37:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:37:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:42' 2024-09-16T13:37:43.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:37:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:43.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:37:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:37:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b923f668-8fc9-4de6-9396-9c071c8843c5 2024-09-16T13:37:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:37:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:37:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:38 2024-09-16T13:37:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:37:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:37:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:42' 2024-09-16T13:37:43.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:37:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:43.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:37:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-16T13:37:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:37:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:44.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:44.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T13:37:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-16T13:37:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:37:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:44.398 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-16T13:37:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:37:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T13:37:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T13:37:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-16T13:37:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:37:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-16T13:37:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:37:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T13:37:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:44.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T13:37:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:37:44.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 081de7ab-65ba-457e-8c2d-6dd87fef7409 2024-09-16T13:37:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43 2024-09-16T13:37:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:37:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43' 2024-09-16T13:37:44.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:37:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:37:44.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:37:44.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:37:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:37:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:44.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:37:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:37:44.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:37:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:37:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 01c77cad-81df-436b-bfc6-ce1e9fd16e59 2024-09-16T13:37:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43 2024-09-16T13:37:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:37:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43' 2024-09-16T13:37:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:44.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:37:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:37:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 55780b8f-37f1-4c9d-99fe-a496489bb95e 2024-09-16T13:37:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43 2024-09-16T13:37:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:37:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43' 2024-09-16T13:37:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:44.479 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:37:44.480 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:37:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:37:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:37:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:37:44.481 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:37:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:37:44.482 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:37:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:37:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:44.483 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:37:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:37:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:37:44.484 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:37:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:37:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:37:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a186c2d5-ec70-413a-a351-4e491091d673 2024-09-16T13:37:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43 2024-09-16T13:37:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:44.485 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:37:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43' 2024-09-16T13:37:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:44.486 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:37:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:37:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:37:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:37:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 03a85e26-8103-4730-a53c-df66946ae4b5 2024-09-16T13:37:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:44.487 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43 2024-09-16T13:37:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:37:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43' 2024-09-16T13:37:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:37:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:37:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:37:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:37:44.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:37:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:37:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:37:44.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:37:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:37:44.491 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:37:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:37:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:37:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:37:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:37:44.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:37:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:37:44.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:37:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:37:44.494 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:37:44.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:44.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-16 13:37:43' 2024-09-16T13:37:44.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:44.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:44.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:37:44.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:44.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:37:43' 2024-09-16T13:37:44.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-16 13:37:43' 2024-09-16T13:37:44.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-16 13:37:43' 2024-09-16T13:37:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-16 13:37:43' 2024-09-16T13:37:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-16 13:37:43' 2024-09-16T13:37:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:37:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:37:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:37:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:37:43' 2024-09-16T13:37:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:44.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:37:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:37:43' 2024-09-16T13:37:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:37:43' 2024-09-16T13:37:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:37:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:37:43' 2024-09-16T13:37:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:44.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:37:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:37:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:37:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:37:44.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43' 2024-09-16T13:37:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:37:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T13:37:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43' 2024-09-16T13:37:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:37:44.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T13:37:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43' 2024-09-16T13:37:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:37:44.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T13:37:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43' 2024-09-16T13:37:44.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:37:44.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T13:37:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43' 2024-09-16T13:37:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:37:44.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T13:37:44.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:44.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T13:37:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:44.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T13:37:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T13:37:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T13:37:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T13:37:44.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:37:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:37:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:37:44.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:37:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:37:44.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-16T13:37:44.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:37:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:37:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:37:44.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:37:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:37:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-16T13:37:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:44.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:37:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:37:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:37:44.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:44.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:37:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-16T13:37:44.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:37:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:44.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:37:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-16T13:37:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:37:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:37:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:44.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-16T13:37:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:37:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:37:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-16T13:37:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:45.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:37:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:45.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T13:37:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-16T13:37:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:37:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:45.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:45.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T13:37:45.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-16T13:37:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:37:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:45.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:45.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T13:37:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-16T13:37:45.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:37:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:45.302 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T13:37:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:37:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: c3613aef-b1a7-426c-babb-f25deca9a330 2024-09-16T13:37:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43 2024-09-16T13:37:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:37:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:45.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43' 2024-09-16T13:37:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:37:45.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:37:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:37:45.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:37:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:45.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:37:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:37:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:37:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:37:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:37:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:37:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: ff91f6f4-8d42-45ca-a2a7-8a79187744cb 2024-09-16T13:37:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43 2024-09-16T13:37:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:37:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43' 2024-09-16T13:37:45.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-16 13:37:43' 2024-09-16T13:37:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:37:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:37:43' 2024-09-16T13:37:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:37:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:37:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:37:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:37:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:37:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:37:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:37:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:37:45.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:37:45.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:37:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:37:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 49499ffb-8600-4c6a-a77d-3131161f82c8 2024-09-16T13:37:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43 2024-09-16T13:37:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:37:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43' 2024-09-16T13:37:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-16 13:37:43' 2024-09-16T13:37:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:37:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:37:43' 2024-09-16T13:37:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:37:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:37:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:37:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:37:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:37:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:37:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:37:45.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:37:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:37:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:37:45.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:45.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-16 13:37:43' 2024-09-16T13:37:45.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:45.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:37:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:37:43' 2024-09-16T13:37:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:45.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:37:45.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:37:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 926c5e9b-1ee1-4c34-af9d-6cb33936c71f 2024-09-16T13:37:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43 2024-09-16T13:37:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:37:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43' 2024-09-16T13:37:45.373 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:37:45.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:37:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:37:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:37:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:37:45.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:45.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:37:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:37:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:37:45.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:37:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-16 13:37:43' 2024-09-16T13:37:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:37:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:37:43' 2024-09-16T13:37:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:45.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:37:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-16T13:37:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:37:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T13:37:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43' 2024-09-16T13:37:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:37:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T13:37:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43' 2024-09-16T13:37:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:37:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T13:37:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:45.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T13:37:45.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:45.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T13:37:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:37:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:37:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:45.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-16T13:37:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:37:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:37:45.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:37:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b923f668-8fc9-4de6-9396-9c071c8843c5 2024-09-16T13:37:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43 2024-09-16T13:37:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:37:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43' 2024-09-16T13:37:45.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:37:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:37:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:37:45.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:37:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:45.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:37:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-16T13:37:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:37:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:37:45.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:37:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:37:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:45.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:37:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:37:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:37:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43' 2024-09-16T13:37:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:37:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:37:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T13:37:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43' 2024-09-16T13:37:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:37:45.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T13:37:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-16 13:37:43' 2024-09-16T13:37:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:37:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:37:45.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:37:43' 2024-09-16T13:37:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:45.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:37:45.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:45.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:45.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T13:37:45.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T13:37:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:37:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:37:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:37:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:45.587 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:37:45.588 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-16T13:37:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:37:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:45.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:37:45.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-16T13:37:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:37:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:45.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:37:45.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:37:43cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:37:43' 2024-09-16T13:37:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:37:45.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T13:37:45.717 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T13:37:45.722 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:37:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:45.723 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:37:45.725 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-16T13:37:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:37:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:45.726 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:37:45.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:37:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-16T13:37:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:37:45.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-16T13:37:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-16T13:37:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:37:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-16T13:37:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:37:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:45.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-16T13:37:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-16T13:37:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:37:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:45.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-16T13:37:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-16T13:37:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-16T13:37:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:37:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:45.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-16T13:37:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:37:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 55780b8f-37f1-4c9d-99fe-a496489bb95e 2024-09-16T13:37:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43 2024-09-16T13:37:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:37:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43' 2024-09-16T13:37:45.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:37:45.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:37:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:37:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:37:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:45.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:37:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:45.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:37:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:37:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:37:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 01c77cad-81df-436b-bfc6-ce1e9fd16e59 2024-09-16T13:37:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43 2024-09-16T13:37:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:37:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43' 2024-09-16T13:37:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:37:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:37:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:37:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:37:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:37:45.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:37:45.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:37:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-16 13:37:43' 2024-09-16T13:37:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:37:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:37:43' 2024-09-16T13:37:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:45.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:37:45.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:37:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:37:45.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:37:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:37:45.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a186c2d5-ec70-413a-a351-4e491091d673 2024-09-16T13:37:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43 2024-09-16T13:37:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:37:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43' 2024-09-16T13:37:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:37:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:37:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 081de7ab-65ba-457e-8c2d-6dd87fef7409 2024-09-16T13:37:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43 2024-09-16T13:37:45.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:37:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43' 2024-09-16T13:37:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:37:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 03a85e26-8103-4730-a53c-df66946ae4b5 2024-09-16T13:37:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43 2024-09-16T13:37:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:37:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:45.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43' 2024-09-16T13:37:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:37:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:37:45.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:37:45.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-16 13:37:43' 2024-09-16T13:37:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:37:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:37:43' 2024-09-16T13:37:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:37:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:37:45.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:37:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:37:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:37:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:37:45.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:37:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:37:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:37:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:45.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:37:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:37:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:37:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:45.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:37:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:37:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:45.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:37:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:37:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:37:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:37:45.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:37:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:37:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:37:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:37:45.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:37:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:37:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:37:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-16 13:37:43' 2024-09-16T13:37:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:37:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:37:43' 2024-09-16T13:37:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-16 13:37:43' 2024-09-16T13:37:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:37:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:37:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:37:43' 2024-09-16T13:37:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:37:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-16 13:37:43' 2024-09-16T13:37:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:37:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:37:43' 2024-09-16T13:37:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:45.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:37:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43' 2024-09-16T13:37:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:37:46.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T13:37:46.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:46.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T13:37:46.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:37:46.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:46.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:37:46.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-16T13:37:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:46.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:37:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:37:46.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-16T13:37:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43' 2024-09-16T13:37:46.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:37:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43' 2024-09-16T13:37:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:37:46.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T13:37:46.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T13:37:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T13:37:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:46.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T13:37:46.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:37:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:37:46.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43' 2024-09-16T13:37:46.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:37:46.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T13:37:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:37:46.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:37:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-16T13:37:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1<+ local image=image-primary4 2024-09-16T13:37:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:37:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mirror_uuids>007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43' 2024-09-16T13:37:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-16T13:37:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:37:46.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-16T13:37:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:37:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:37:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-16T13:37:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T13:37:46.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T13:37:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:37:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:37:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T13:37:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-16T13:37:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:37:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:37:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-16T13:37:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:37:46.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:37:46.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:46.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:46.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-16T13:37:46.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:46.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:46.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:37:46.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:37:46.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-16T13:37:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-16T13:37:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:37:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-16T13:37:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-16T13:37:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:37:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:46.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-16T13:37:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-16T13:37:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:46.589 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:37:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:46.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-16T13:37:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:37:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-16T13:37:46.598 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:37:46.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:37:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-16T13:37:46.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:37:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:37:46.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:37:46.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:37:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:37:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:37:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary10 2024-09-16T13:37:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:37:46.635 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:37:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary1 2024-09-16T13:37:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary4 2024-09-16T13:37:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary6 2024-09-16T13:37:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary9 2024-09-16T13:37:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:37:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: ff91f6f4-8d42-45ca-a2a7-8a79187744cb 2024-09-16T13:37:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43 2024-09-16T13:37:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:37:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43' 2024-09-16T13:37:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:37:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:37:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:37:46.667 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:37:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:46.668 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:37:46.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:46.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:37:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:37:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: c3613aef-b1a7-426c-babb-f25deca9a330 2024-09-16T13:37:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43 2024-09-16T13:37:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:37:46.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43' 2024-09-16T13:37:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:46.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:37:46.690 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:37:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:37:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:37:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:37:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:37:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 49499ffb-8600-4c6a-a77d-3131161f82c8 2024-09-16T13:37:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43 2024-09-16T13:37:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:37:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43' 2024-09-16T13:37:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:37:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:37:46.701 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:37:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:37:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:46.702 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:37:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:46.703 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:37:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:37:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 926c5e9b-1ee1-4c34-af9d-6cb33936c71f 2024-09-16T13:37:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43 2024-09-16T13:37:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:37:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:46.706 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43' 2024-09-16T13:37:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:37:46.707 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:37:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:37:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:37:46.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:37:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:37:46.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:37:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:37:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:37:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:37:46.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:46.711 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:37:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-16 13:37:43' 2024-09-16T13:37:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:37:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:46.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:37:43' 2024-09-16T13:37:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-16 13:37:43' 2024-09-16T13:37:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:37:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:37:43' 2024-09-16T13:37:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:46.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:37:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:37:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:37:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:37:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:37:46.715 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:37:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:46.718 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:37:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-16 13:37:43' 2024-09-16T13:37:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:37:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:37:43' 2024-09-16T13:37:46.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-16T13:37:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:37:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:37:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:37:46.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:46.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-16T13:37:46.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:37:46.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:37:46.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:46.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:37:46.741 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:37:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-16 13:37:43' 2024-09-16T13:37:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:37:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:37:43' 2024-09-16T13:37:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:46.755 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:37:46.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd0 2024-09-16T13:37:46.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:37:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:37:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b923f668-8fc9-4de6-9396-9c071c8843c5 2024-09-16T13:37:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:37:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43 2024-09-16T13:37:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:37:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:37:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:37:46.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:37:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:37:43' 2024-09-16T13:37:46.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:37:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd0 2024-09-16T13:37:46.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:37:46.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:37:46.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:37:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:37:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:37:46.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:46.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:46.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:37:46.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:37:46.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:37:46.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:37:46.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd1 2024-09-16T13:37:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43' 2024-09-16T13:37:46.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:37:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd2 2024-09-16T13:37:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T13:37:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-16 13:37:43' 2024-09-16T13:37:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:37:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:37:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:37:43' 2024-09-16T13:37:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:37:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:37:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:37:46.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd1 2024-09-16T13:37:46.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd2 2024-09-16T13:37:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:37:46.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:37:46.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T13:37:46.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:37:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:37:46.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:46.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:46.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-16T13:37:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:37:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:37:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-16T13:37:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd3 2024-09-16T13:37:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:37:46.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd3 2024-09-16T13:37:46.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd4 2024-09-16T13:37:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:37:46.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd4 2024-09-16T13:37:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43' 2024-09-16T13:37:46.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:37:47.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T13:37:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T13:37:47.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:37:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:37:47.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-16T13:37:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:37:47.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:37:47.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-16T13:37:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43' 2024-09-16T13:37:47.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:37:47.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T13:37:47.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T13:37:47.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:37:47.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:37:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:47.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-16T13:37:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1<+ local image=image-primary5 2024-09-16T13:37:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:37:47.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mirror_uuids>007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43' 2024-09-16T13:37:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-16T13:37:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:37:47.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T13:37:47.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:47.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T13:37:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:37:47.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:37:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:47.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-16T13:37:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:37:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:37:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-16T13:37:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:37:43cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:37:43' 2024-09-16T13:37:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:37:47.202 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T13:37:47.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:37:43' 2024-09-16T13:37:47.209 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T13:37:47.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:37:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:37:47.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:37:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:37:47.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:47.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:37:47.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-16T13:37:47.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:37:47.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:37:47.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:37:47.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:37:47.222 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-16T13:37:47.605 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:37:47.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:37:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:37:47.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:37:47.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:37:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:37:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:37:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary7 2024-09-16T13:37:47.643 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary8 2024-09-16T13:37:47.644 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:37:47.645 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary5 2024-09-16T13:37:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:37:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary3 2024-09-16T13:37:47.659 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:37:47.660 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary2 2024-09-16T13:37:47.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd5 2024-09-16T13:37:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd5 2024-09-16T13:37:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:37:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd6 2024-09-16T13:37:47.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:37:47.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd6 2024-09-16T13:37:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd7 2024-09-16T13:37:48.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd7 2024-09-16T13:37:48.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:37:48.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd8 2024-09-16T13:37:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:37:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd8 2024-09-16T13:37:48.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd9 2024-09-16T13:37:48.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:37:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd9 2024-09-16T13:37:50.061 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:37:50.051+0000 7ff4f6946700 -1 reset not still connected to 0x55756b66c9c0 2024-09-16T13:37:50.062 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:37:50.051+0000 7ff4f6946700 -1 reset not still connected to 0x55756d08ac30 2024-09-16T13:37:50.062 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:37:50.051+0000 7ff4f6946700 -1 reset not still connected to 0x557574d54820 2024-09-16T13:37:50.766 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:37:50.753+0000 7f74def64700 -1 reset not still connected to 0x559465f19c70 2024-09-16T13:37:51.459 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:37:51.445+0000 7f315ffe6700 -1 reset not still connected to 0x55c1fb1aba00 2024-09-16T13:39:21.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=9b234a348fa17bc0a853589d3a349ba8 2024-09-16T13:39:21.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd4 2024-09-16T13:39:21.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=3a56e2b07dcf980648d5a577d85cae12 2024-09-16T13:39:21.534 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-09-16T13:39:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=8ba0351299f182d21361b09ccf60429b 2024-09-16T13:39:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd2 2024-09-16T13:39:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=1e185adf23d636df7c4627a009087a0d 2024-09-16T13:39:22.665 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd1 2024-09-16T13:39:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 3a56e2b07dcf980648d5a577d85cae12 != \3\a\5\6\e\2\b\0\7\d\c\f\9\8\0\6\4\8\d\5\a\5\7\7\d\8\5\c\a\e\1\2 ]] 2024-09-16T13:39:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 8ba0351299f182d21361b09ccf60429b != \8\b\a\0\3\5\1\2\9\9\f\1\8\2\d\2\1\3\6\1\b\0\9\c\c\f\6\0\4\2\9\b ]] 2024-09-16T13:39:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 9b234a348fa17bc0a853589d3a349ba8 != \9\b\2\3\4\a\3\4\8\f\a\1\7\b\c\0\a\8\5\3\5\8\9\d\3\a\3\4\9\b\a\8 ]] 2024-09-16T13:39:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 1e185adf23d636df7c4627a009087a0d != \1\e\1\8\5\a\d\f\2\3\d\6\3\6\d\f\7\c\4\6\2\7\a\0\0\9\0\8\7\a\0\d ]] 2024-09-16T13:39:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:39:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 128166 2024-09-16T13:39:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=e79e73bad3cccff91074d8413d336cf7 2024-09-16T13:39:24.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd3 2024-09-16T13:39:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=644922d124c736a1e2a7c64e41c88f87 2024-09-16T13:39:24.637 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd7 2024-09-16T13:39:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=5742100030069f1f3634fe7e07047941 2024-09-16T13:39:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd9 2024-09-16T13:39:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=476362081063b57a5f938436fbfd5330 2024-09-16T13:39:24.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd6 2024-09-16T13:39:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=f01371c5506c32e1bd3308fd65292ca5 2024-09-16T13:39:24.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd8 2024-09-16T13:39:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=1551a5ac1f610658954cb17610a962a7 2024-09-16T13:39:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd5 2024-09-16T13:39:25.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ e79e73bad3cccff91074d8413d336cf7 != \e\7\9\e\7\3\b\a\d\3\c\c\c\f\f\9\1\0\7\4\d\8\4\1\3\d\3\3\6\c\f\7 ]] 2024-09-16T13:39:25.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 644922d124c736a1e2a7c64e41c88f87 != \6\4\4\9\2\2\d\1\2\4\c\7\3\6\a\1\e\2\a\7\c\6\4\e\4\1\c\8\8\f\8\7 ]] 2024-09-16T13:39:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 476362081063b57a5f938436fbfd5330 != \4\7\6\3\6\2\0\8\1\0\6\3\b\5\7\a\5\f\9\3\8\4\3\6\f\b\f\d\5\3\3\0 ]] 2024-09-16T13:39:25.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 5742100030069f1f3634fe7e07047941 != \5\7\4\2\1\0\0\0\3\0\0\6\9\f\1\f\3\6\3\4\f\e\7\e\0\7\0\4\7\9\4\1 ]] 2024-09-16T13:39:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ f01371c5506c32e1bd3308fd65292ca5 != \f\0\1\3\7\1\c\5\5\0\6\c\3\2\e\1\b\d\3\3\0\8\f\d\6\5\2\9\2\c\a\5 ]] 2024-09-16T13:39:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:39:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 128167 2024-09-16T13:39:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:39:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 128168 2024-09-16T13:39:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:39:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 128171 2024-09-16T13:39:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:39:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 128174 2024-09-16T13:39:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:39:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 128176 2024-09-16T13:39:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:39:25.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 128177 2024-09-16T13:39:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 1551a5ac1f610658954cb17610a962a7 != \1\5\5\1\a\5\a\c\1\f\6\1\0\6\5\8\9\5\4\c\b\1\7\6\1\0\a\9\6\2\a\7 ]] 2024-09-16T13:39:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:39:25.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 128178 2024-09-16T13:39:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:39:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 128180 2024-09-16T13:39:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T13:39:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:39:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary1 2024-09-16T13:39:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-16T13:39:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:39:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:39:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:39:25.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:39:25.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:39:26.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-16T13:39:26.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:26.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:26.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:39:26.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:26.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:39:26.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:39:26.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:39:26.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:39:26.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T13:39:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:39:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: a186c2d5-ec70-413a-a351-4e491091d673 2024-09-16T13:39:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:39:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":98140160.0,"last_snapshot_bytes":98140160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"} 2024-09-16T13:39:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:39:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:39:23 2024-09-16T13:39:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:39:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:39:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:39:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:39:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:39:23' 2024-09-16T13:39:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:39:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:39:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:39:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":98140160.0,"last_snapshot_bytes":98140160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"} 2024-09-16T13:39:27.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:39:27.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:39:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:39:27.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:39:27.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:39:27.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:39:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:39:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:39:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:39:23' 2024-09-16T13:39:27.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-16 13:39:23' 2024-09-16T13:39:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:39:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:39:23' 2024-09-16T13:39:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:39:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:39:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:39:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1a186c2d5-ec70-413a-a351-4e491091d673up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":98140160.0,"last_snapshot_bytes":98140160,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:23image-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:23image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70881280.0,"last_snapshot_bytes":70881280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:23image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72204288.0,"last_snapshot_bytes":72204288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:23image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72359936.0,"last_snapshot_bytes":72359936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:23image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70123520.0,"last_snapshot_bytes":70123520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:23image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":129359872.0,"last_snapshot_bytes":129359872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:23image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70209536.0,"last_snapshot_bytes":70209536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:23image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70148096.0,"last_snapshot_bytes":70148096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:23image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60436480.0,"last_snapshot_bytes":60436480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:23' 2024-09-16T13:39:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:39:27.617 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T13:39:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:39:23' 2024-09-16T13:39:27.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T13:39:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:39:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:39:27.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:39:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:39:27.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:27.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:27.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-16T13:39:27.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:39:27.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:27.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:39:27.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-16T13:39:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-16T13:39:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:39:29.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-16T13:39:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:39:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:39:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:39:29.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary1 2024-09-16T13:39:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:30.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary1 2024-09-16T13:39:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:39:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary2 2024-09-16T13:39:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-16T13:39:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:30.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:39:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:39:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:39:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:39:30.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:39:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-16T13:39:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:39:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:39:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:39:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:39:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:39:31.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T13:39:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:39:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b923f668-8fc9-4de6-9396-9c071c8843c5 2024-09-16T13:39:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:39:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70881280.0,"last_snapshot_bytes":70881280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"} 2024-09-16T13:39:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:39:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:39:23 2024-09-16T13:39:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:39:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:39:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:39:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:39:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:39:28' 2024-09-16T13:39:31.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:39:31.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:39:31.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:39:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70881280.0,"last_snapshot_bytes":70881280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"} 2024-09-16T13:39:31.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:39:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:39:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:39:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:39:31.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:39:31.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:39:31.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:39:31.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:39:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:39:23' 2024-09-16T13:39:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-16 13:39:23' 2024-09-16T13:39:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:39:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:39:23' 2024-09-16T13:39:31.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:39:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:39:31.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:39:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK942544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:28image-primary2b923f668-8fc9-4de6-9396-9c071c8843c5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70881280.0,"last_snapshot_bytes":70881280,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:28image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72204288.0,"last_snapshot_bytes":72204288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:28image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72359936.0,"last_snapshot_bytes":72359936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:28image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70123520.0,"last_snapshot_bytes":70123520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:28image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":129359872.0,"last_snapshot_bytes":129359872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:28image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70209536.0,"last_snapshot_bytes":70209536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:28image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70148096.0,"last_snapshot_bytes":70148096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:28image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60436480.0,"last_snapshot_bytes":60436480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:28' 2024-09-16T13:39:31.502 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:39:31.503 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T13:39:31.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:39:23' 2024-09-16T13:39:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T13:39:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:39:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:39:31.518 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:39:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:39:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-16T13:39:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:39:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:39:31.521 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-16T13:39:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-16T13:39:33.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:39:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-16T13:39:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:39:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:39:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:39:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary2 2024-09-16T13:39:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary2 2024-09-16T13:39:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:39:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary3 2024-09-16T13:39:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-16T13:39:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:39:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:39:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:39:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:39:34.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:39:35.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-16T13:39:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:39:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:39:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:39:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:39:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:39:35.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T13:39:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:39:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: ff91f6f4-8d42-45ca-a2a7-8a79187744cb 2024-09-16T13:39:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:39:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72204288.0,"last_snapshot_bytes":72204288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"} 2024-09-16T13:39:35.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:39:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:39:23 2024-09-16T13:39:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:39:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:39:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:39:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:39:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:39:33' 2024-09-16T13:39:35.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:39:35.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:39:35.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:39:35.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72204288.0,"last_snapshot_bytes":72204288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"} 2024-09-16T13:39:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:39:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:39:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:39:35.132 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:39:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:39:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:39:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:39:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:39:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:39:23' 2024-09-16T13:39:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-16 13:39:23' 2024-09-16T13:39:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:39:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:39:23' 2024-09-16T13:39:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:39:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:39:35.141 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:39:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK842544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:33image-primary3ff91f6f4-8d42-45ca-a2a7-8a79187744cbup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72204288.0,"last_snapshot_bytes":72204288,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:33image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72359936.0,"last_snapshot_bytes":72359936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:33image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70123520.0,"last_snapshot_bytes":70123520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:33image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":129359872.0,"last_snapshot_bytes":129359872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:33image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70209536.0,"last_snapshot_bytes":70209536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:33image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70148096.0,"last_snapshot_bytes":70148096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:33image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60436480.0,"last_snapshot_bytes":60436480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:33' 2024-09-16T13:39:35.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:39:35.325 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T13:39:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:39:23' 2024-09-16T13:39:35.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T13:39:35.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:39:35.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:39:35.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:39:35.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:39:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-16T13:39:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:39:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:39:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-16T13:39:35.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-16T13:39:37.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:39:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-16T13:39:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:37.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:39:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:39:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:39:37.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary3 2024-09-16T13:39:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:37.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary3 2024-09-16T13:39:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:39:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary4 2024-09-16T13:39:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-16T13:39:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:39:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:39:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:39:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:39:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:39:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-16T13:39:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:39:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:39:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:39:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:39:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:39:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T13:39:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:39:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 01c77cad-81df-436b-bfc6-ce1e9fd16e59 2024-09-16T13:39:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:39:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72359936.0,"last_snapshot_bytes":72359936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"} 2024-09-16T13:39:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:39:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:39:23 2024-09-16T13:39:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:39:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:39:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:39:38.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:39:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:39:38' 2024-09-16T13:39:38.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:39:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:39:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:39:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72359936.0,"last_snapshot_bytes":72359936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"} 2024-09-16T13:39:38.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:39:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:39:38.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:39:38.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:39:38.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:39:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:39:38.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:39:38.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:39:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:39:23' 2024-09-16T13:39:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-16 13:39:23' 2024-09-16T13:39:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:39:38.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:39:23' 2024-09-16T13:39:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:39:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:39:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:39:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK742544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:38image-primary401c77cad-81df-436b-bfc6-ce1e9fd16e59up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72359936.0,"last_snapshot_bytes":72359936,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:38image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70123520.0,"last_snapshot_bytes":70123520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:38image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":129359872.0,"last_snapshot_bytes":129359872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:38image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70209536.0,"last_snapshot_bytes":70209536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:38image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70148096.0,"last_snapshot_bytes":70148096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:38image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60436480.0,"last_snapshot_bytes":60436480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:38' 2024-09-16T13:39:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:39:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T13:39:39.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:39:23' 2024-09-16T13:39:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T13:39:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:39:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:39:39.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:39:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:39:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-16T13:39:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:39:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:39:39.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-16T13:39:39.206 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-16T13:39:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:39:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-16T13:39:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:39:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:39:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:39:40.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary4 2024-09-16T13:39:40.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:40.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary4 2024-09-16T13:39:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:39:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary5 2024-09-16T13:39:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-16T13:39:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:39:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:39:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:39:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:39:40.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:39:40.990 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:39:40.981+0000 7f61b1855700 -1 reset not still connected to 0x557b1f47fd40 2024-09-16T13:39:40.990 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:39:40.981+0000 7f61b1855700 -1 reset not still connected to 0x557b229664e0 2024-09-16T13:39:40.990 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:39:40.981+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T13:39:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-16T13:39:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:39:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:39:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:39:41.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:39:41.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:39:41.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T13:39:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:39:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: c3613aef-b1a7-426c-babb-f25deca9a330 2024-09-16T13:39:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:39:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70123520.0,"last_snapshot_bytes":70123520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"} 2024-09-16T13:39:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:39:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:39:23 2024-09-16T13:39:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:39:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:39:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:39:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:39:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:39:38' 2024-09-16T13:39:42.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:39:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:39:42.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:39:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70123520.0,"last_snapshot_bytes":70123520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"} 2024-09-16T13:39:42.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:39:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:39:42.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:39:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:39:42.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:39:42.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:39:42.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:39:42.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:39:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:39:23' 2024-09-16T13:39:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-16 13:39:23' 2024-09-16T13:39:42.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:39:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:39:23' 2024-09-16T13:39:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:39:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:39:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:39:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK642544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:38image-primary5c3613aef-b1a7-426c-babb-f25deca9a330up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70123520.0,"last_snapshot_bytes":70123520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:38image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":129359872.0,"last_snapshot_bytes":129359872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:38image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70209536.0,"last_snapshot_bytes":70209536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:38image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70148096.0,"last_snapshot_bytes":70148096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:38image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60436480.0,"last_snapshot_bytes":60436480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:38' 2024-09-16T13:39:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:39:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T13:39:42.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:39:23' 2024-09-16T13:39:42.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T13:39:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:39:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:39:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:39:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:39:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-16T13:39:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:39:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:39:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-16T13:39:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-16T13:39:44.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:39:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-16T13:39:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:44.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:39:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:39:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:39:44.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary5 2024-09-16T13:39:44.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary5 2024-09-16T13:39:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:39:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary6 2024-09-16T13:39:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-16T13:39:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:39:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:39:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:39:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:39:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:39:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-16T13:39:45.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:39:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:39:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:39:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:39:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:39:45.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T13:39:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:39:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 081de7ab-65ba-457e-8c2d-6dd87fef7409 2024-09-16T13:39:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:39:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":129359872.0,"last_snapshot_bytes":129359872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"} 2024-09-16T13:39:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:39:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:39:23 2024-09-16T13:39:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:39:46.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:39:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:39:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:39:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:39:43' 2024-09-16T13:39:46.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:39:46.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:39:46.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:39:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":129359872.0,"last_snapshot_bytes":129359872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"} 2024-09-16T13:39:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:39:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:39:46.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:39:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:39:46.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:39:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:39:46.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:39:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:39:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:39:23' 2024-09-16T13:39:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-16 13:39:23' 2024-09-16T13:39:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:46.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:39:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:39:23' 2024-09-16T13:39:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:39:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:39:46.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:39:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK542544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:43image-primary6081de7ab-65ba-457e-8c2d-6dd87fef7409up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":129359872.0,"last_snapshot_bytes":129359872,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:43image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70209536.0,"last_snapshot_bytes":70209536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:43image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70148096.0,"last_snapshot_bytes":70148096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:43image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60436480.0,"last_snapshot_bytes":60436480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:43' 2024-09-16T13:39:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:39:46.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T13:39:46.216 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:39:23' 2024-09-16T13:39:46.217 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T13:39:46.223 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:39:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:39:46.224 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:39:46.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:39:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-16T13:39:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:39:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:39:46.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-16T13:39:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-16T13:39:47.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:39:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-16T13:39:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:47.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:39:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:39:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:39:47.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary6 2024-09-16T13:39:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:47.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary6 2024-09-16T13:39:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:39:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary7 2024-09-16T13:39:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-16T13:39:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:39:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:39:47.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:39:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:39:47.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:39:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-16T13:39:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:39:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:39:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:39:48.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:39:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:39:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T13:39:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:39:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 926c5e9b-1ee1-4c34-af9d-6cb33936c71f 2024-09-16T13:39:49.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:39:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70209536.0,"last_snapshot_bytes":70209536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"} 2024-09-16T13:39:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:39:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:39:23 2024-09-16T13:39:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:39:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:39:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:39:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:39:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:39:48' 2024-09-16T13:39:49.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:39:49.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:39:49.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:39:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70209536.0,"last_snapshot_bytes":70209536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"} 2024-09-16T13:39:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:39:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:39:49.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:39:49.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:39:49.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:39:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:39:49.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:39:49.049 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:39:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:39:23' 2024-09-16T13:39:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-16 13:39:23' 2024-09-16T13:39:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:39:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:39:23' 2024-09-16T13:39:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:39:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:39:49.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:39:49.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK442544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:48image-primary7926c5e9b-1ee1-4c34-af9d-6cb33936c71fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70209536.0,"last_snapshot_bytes":70209536,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:48image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70148096.0,"last_snapshot_bytes":70148096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:48image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60436480.0,"last_snapshot_bytes":60436480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:48' 2024-09-16T13:39:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:39:49.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T13:39:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:39:23' 2024-09-16T13:39:49.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T13:39:49.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:39:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:39:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:39:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:39:49.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-16T13:39:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:39:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:39:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-16T13:39:49.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-16T13:39:50.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:39:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-16T13:39:50.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:39:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:39:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary7 2024-09-16T13:39:50.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:39:50.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:50.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary7 2024-09-16T13:39:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:39:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary8 2024-09-16T13:39:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-16T13:39:50.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:39:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:39:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:39:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:39:50.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:39:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-16T13:39:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:51.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:39:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:39:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:39:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:39:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:39:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T13:39:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:39:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 49499ffb-8600-4c6a-a77d-3131161f82c8 2024-09-16T13:39:52.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:39:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70148096.0,"last_snapshot_bytes":70148096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"} 2024-09-16T13:39:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:39:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:39:23 2024-09-16T13:39:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:39:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:39:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:39:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:39:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:39:48' 2024-09-16T13:39:52.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:39:52.078 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:39:52.079 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:39:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70148096.0,"last_snapshot_bytes":70148096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"} 2024-09-16T13:39:52.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:39:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:39:52.082 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:39:52.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:39:52.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:39:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:39:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:39:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:39:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:39:23' 2024-09-16T13:39:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-16 13:39:23' 2024-09-16T13:39:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:39:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:39:23' 2024-09-16T13:39:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:39:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:39:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:39:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK342544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:48image-primary849499ffb-8600-4c6a-a77d-3131161f82c8up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70148096.0,"last_snapshot_bytes":70148096,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493867,"remote_snapshot_timestamp":1726493867,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:48image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60436480.0,"last_snapshot_bytes":60436480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:48' 2024-09-16T13:39:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:39:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T13:39:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:39:23' 2024-09-16T13:39:52.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T13:39:52.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:39:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:39:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:39:52.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:39:52.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:52.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:52.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-16T13:39:52.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:39:52.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:52.244 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:39:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-16T13:39:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-16T13:39:53.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:39:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-16T13:39:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:39:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:39:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:39:53.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary8 2024-09-16T13:39:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:53.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary8 2024-09-16T13:39:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:39:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary9 2024-09-16T13:39:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-16T13:39:53.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:39:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:39:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:39:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:39:53.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:39:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-16T13:39:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:39:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:54.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:39:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:39:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:39:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:39:54.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T13:39:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:39:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 03a85e26-8103-4730-a53c-df66946ae4b5 2024-09-16T13:39:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:39:55.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60436480.0,"last_snapshot_bytes":60436480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"} 2024-09-16T13:39:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:39:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:39:53 2024-09-16T13:39:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:39:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:39:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:39:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:39:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:39:53' 2024-09-16T13:39:55.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:39:55.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:39:55.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:39:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60436480.0,"last_snapshot_bytes":60436480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"} 2024-09-16T13:39:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:39:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:39:55.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:39:55.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:39:55.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:39:55.093 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:39:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:39:55.094 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:39:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:39:53' 2024-09-16T13:39:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-16 13:39:53' 2024-09-16T13:39:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:39:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:39:53' 2024-09-16T13:39:55.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:39:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:39:55.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:39:55.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK242544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:53image-primary903a85e26-8103-4730-a53c-df66946ae4b5up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":60436480.0,"last_snapshot_bytes":60436480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:53' 2024-09-16T13:39:55.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:39:55.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T13:39:55.204 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:39:53' 2024-09-16T13:39:55.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T13:39:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:39:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:39:55.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:39:55.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:39:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-16T13:39:55.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:39:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:39:55.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-16T13:39:55.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-16T13:39:56.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:39:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-16T13:39:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:39:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:39:56.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:39:56.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary9 2024-09-16T13:39:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:56.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary9 2024-09-16T13:39:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:39:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary10 2024-09-16T13:39:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-16T13:39:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:56.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:39:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:39:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:39:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:39:56.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:39:57.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-16T13:39:57.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:57.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:57.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:39:57.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:57.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:39:57.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:39:57.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:39:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:39:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T13:39:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:39:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 55780b8f-37f1-4c9d-99fe-a496489bb95e 2024-09-16T13:39:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:39:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"} 2024-09-16T13:39:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:39:57.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:39:53 2024-09-16T13:39:57.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:39:57.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:39:57.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:39:57.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:39:57.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:39:53' 2024-09-16T13:39:57.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:39:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:39:57.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:39:57.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"} 2024-09-16T13:39:57.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:39:57.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:39:57.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:39:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:39:57.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:39:57.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:39:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:39:57.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:39:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:39:53' 2024-09-16T13:39:57.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-16 13:39:53' 2024-09-16T13:39:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:39:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:39:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:39:53' 2024-09-16T13:39:57.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:39:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:39:57.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:39:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK142544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1055780b8f-37f1-4c9d-99fe-a496489bb95eup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726493866,"remote_snapshot_timestamp":1726493866,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:39:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:39:53' 2024-09-16T13:39:58.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:39:58.081 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T13:39:58.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:39:53' 2024-09-16T13:39:58.089 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T13:39:58.095 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:39:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:39:58.096 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:39:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:39:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-16T13:39:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:39:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:39:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-16T13:39:58.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-16T13:39:59.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:39:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-16T13:39:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:39:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:39:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:39:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary10 2024-09-16T13:39:59.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:39:59.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary10 2024-09-16T13:39:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..10} 2024-09-16T13:39:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS=() 2024-09-16T13:39:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS=() 2024-09-16T13:39:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS=() 2024-09-16T13:39:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS=() 2024-09-16T13:39:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ RET=0 2024-09-16T13:39:59.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:39:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary1 2024-09-16T13:39:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary1 2024-09-16T13:39:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary1 snapshot 10G 2024-09-16T13:39:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:39:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:39:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:39:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:39:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:39:59.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:40:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:40:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary1 10G 2024-09-16T13:40:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:40:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:40:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:40:00.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:40:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:00.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary1 2024-09-16T13:40:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:40:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary1 snapshot 2024-09-16T13:40:00.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:40:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:40:00.091 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary1 snapshot 2024-09-16T13:40:00.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:40:00.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary1 2024-09-16T13:40:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary1': 2024-09-16T13:40:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:40:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:40:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:40:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 3a2e2438b5e1 2024-09-16T13:40:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.3a2e2438b5e1 2024-09-16T13:40:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:40:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:40:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:40:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:40:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:40:00 2024 2024-09-16T13:40:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:40:00 2024 2024-09-16T13:40:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:40:00 2024 2024-09-16T13:40:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:40:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:40:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: ef925c50-dcd8-411f-9652-6d0180c073f0 2024-09-16T13:40:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:40:00.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:40:00.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary1 2024-09-16T13:40:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd0 2024-09-16T13:40:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:40:01.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd0 2024-09-16T13:40:01.090 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:40:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:40:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:40:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 3bbf1467-39c3-4df7-99a5-440308bdb1bd 2024-09-16T13:40:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:40:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:40:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:40:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:40:01.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:40:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:40:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:40:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:40:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary1 2024-09-16T13:40:01.698 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd0 test-primary1 2024-09-16T13:40:01.744 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:40:01.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary1 2024-09-16T13:40:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:40:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary1 2024-09-16T13:40:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:40:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:40:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary1 2024-09-16T13:40:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:40:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:40:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:01.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:40:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:40:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary1 2024-09-16T13:40:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary1 2024-09-16T13:40:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary2 2024-09-16T13:40:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary2 2024-09-16T13:40:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary2 snapshot 10G 2024-09-16T13:40:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:40:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:01.762 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:40:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:40:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary2 10G 2024-09-16T13:40:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:40:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:40:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:40:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:40:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:01.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary2 2024-09-16T13:40:01.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:40:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary2 snapshot 2024-09-16T13:40:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:40:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:40:01.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary2 snapshot 2024-09-16T13:40:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:40:01.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary1/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary1' 2024-09-16T13:40:02.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5943 2024-09-16T13:40:02.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:40:02.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:02.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary2 2024-09-16T13:40:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary2': 2024-09-16T13:40:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:40:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:40:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:40:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 3a3e3906716d 2024-09-16T13:40:02.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.3a3e3906716d 2024-09-16T13:40:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:40:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:40:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:40:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:40:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:40:01 2024 2024-09-16T13:40:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:40:01 2024 2024-09-16T13:40:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:40:01 2024 2024-09-16T13:40:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:40:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:40:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 4c7bec10-21b0-4c4a-9fcd-9eafb063806b 2024-09-16T13:40:02.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:40:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:40:02.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary2 2024-09-16T13:40:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd1 2024-09-16T13:40:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:40:03.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd1 2024-09-16T13:40:03.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:40:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:40:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:40:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 480e6936-96c2-4205-9275-864f7b43b77d 2024-09-16T13:40:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:40:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:40:03.104 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:40:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:40:03.105 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:40:03.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:40:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:40:03.489 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:40:03.490 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary2 2024-09-16T13:40:03.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd1 test-primary2 2024-09-16T13:40:03.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:40:03.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary2 2024-09-16T13:40:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:40:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary2 2024-09-16T13:40:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:40:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:40:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:40:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:40:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:40:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:40:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary2 2024-09-16T13:40:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary3 2024-09-16T13:40:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary3 2024-09-16T13:40:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary3 snapshot 10G 2024-09-16T13:40:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary2 2024-09-16T13:40:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary2 2024-09-16T13:40:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:40:03.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:40:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:40:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary3 10G 2024-09-16T13:40:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:03.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:40:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:40:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:40:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:40:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:03.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary3 2024-09-16T13:40:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:40:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary3 snapshot 2024-09-16T13:40:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:40:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:40:03.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary3 snapshot 2024-09-16T13:40:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:40:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary2/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary2' 2024-09-16T13:40:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5947 2024-09-16T13:40:03.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:04.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:40:04.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary3 2024-09-16T13:40:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary3': 2024-09-16T13:40:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:40:04.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:40:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:40:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 3a4da87bf790 2024-09-16T13:40:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.3a4da87bf790 2024-09-16T13:40:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:40:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:40:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:40:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:40:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:40:03 2024 2024-09-16T13:40:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:40:03 2024 2024-09-16T13:40:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:40:03 2024 2024-09-16T13:40:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:40:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:40:04.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: bf98e067-5114-462a-827a-e4b70ac5d828 2024-09-16T13:40:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:40:04.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:40:04.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary3 2024-09-16T13:40:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd2 2024-09-16T13:40:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:40:05.083 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd2 2024-09-16T13:40:05.092 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:40:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:40:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:40:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: fc809d39-946c-4e7e-9f62-c0173a830e2b 2024-09-16T13:40:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:40:05.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:40:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:40:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:40:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:40:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:40:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:40:05.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:40:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary3 2024-09-16T13:40:05.776 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd2 test-primary3 2024-09-16T13:40:05.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:05.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:40:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:40:05.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:40:05.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:40:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary3 2024-09-16T13:40:05.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:40:05.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary3 2024-09-16T13:40:05.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:05.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:40:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:05.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:40:05.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:40:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:40:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:40:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:40:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary4 2024-09-16T13:40:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary4 2024-09-16T13:40:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary4 snapshot 10G 2024-09-16T13:40:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary3 2024-09-16T13:40:05.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:05.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:05.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:05.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:05.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:40:05.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:05.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:40:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary3 2024-09-16T13:40:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:40:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary3 2024-09-16T13:40:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary4 10G 2024-09-16T13:40:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:40:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:40:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:40:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:40:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary4 2024-09-16T13:40:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:40:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary4 snapshot 2024-09-16T13:40:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:40:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:40:05.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary4 snapshot 2024-09-16T13:40:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:40:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary3/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary3' 2024-09-16T13:40:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:40:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:40:06.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:40:06.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5951 2024-09-16T13:40:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:40:06.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary4 2024-09-16T13:40:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary4': 2024-09-16T13:40:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:40:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:40:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:40:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 3a5d1d70e032 2024-09-16T13:40:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.3a5d1d70e032 2024-09-16T13:40:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:40:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:40:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:40:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:40:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:40:05 2024 2024-09-16T13:40:06.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:40:05 2024 2024-09-16T13:40:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:40:05 2024 2024-09-16T13:40:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:40:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:40:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 9f5f2075-de41-4691-9994-66fe545d5690 2024-09-16T13:40:06.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:40:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:40:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary4 2024-09-16T13:40:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd3 2024-09-16T13:40:07.143 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:40:07.144 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd3 2024-09-16T13:40:07.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:40:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:40:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:40:07.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: f177989c-4d4c-4af7-b470-ee4ab32c496d 2024-09-16T13:40:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:40:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:40:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:40:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:40:07.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:40:07.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:40:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 Snapshot ID: 5950 2024-09-16T13:40:07.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T13:40:07.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:40:07.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary4 2024-09-16T13:40:07.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd3 test-primary4 2024-09-16T13:40:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:40:08.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary4 2024-09-16T13:40:08.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:40:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary4 2024-09-16T13:40:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:40:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:40:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:40:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:40:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:40:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:40:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary5 2024-09-16T13:40:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary5 2024-09-16T13:40:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary4 2024-09-16T13:40:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary5 snapshot 10G 2024-09-16T13:40:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary4 2024-09-16T13:40:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary4 2024-09-16T13:40:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:40:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:40:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:40:08.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:08.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary5 10G 2024-09-16T13:40:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:40:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:40:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:40:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:40:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:08.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary5 2024-09-16T13:40:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:40:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary5 snapshot 2024-09-16T13:40:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:40:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:40:08.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary5 snapshot 2024-09-16T13:40:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:40:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary4/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary4' 2024-09-16T13:40:08.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5954 2024-09-16T13:40:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:08.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5956 2024-09-16T13:40:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:40:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:08.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary5 2024-09-16T13:40:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary5': 2024-09-16T13:40:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:40:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:40:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:40:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 3a6f893368a9 2024-09-16T13:40:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.3a6f893368a9 2024-09-16T13:40:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:40:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:40:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:40:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:40:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:40:08 2024 2024-09-16T13:40:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:40:08 2024 2024-09-16T13:40:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:40:08 2024 2024-09-16T13:40:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:40:08.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:40:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 6a8d0386-ae02-4545-9d78-161debb7a66b 2024-09-16T13:40:08.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:40:08.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:40:08.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary5 2024-09-16T13:40:09.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd4 2024-09-16T13:40:09.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:40:09.080 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd4 2024-09-16T13:40:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:40:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:40:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:40:09.174 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: c6a9b607-5353-4974-96cd-5d5cf0bb775c 2024-09-16T13:40:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:40:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:40:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:40:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:40:09.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:40:09.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:40:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:40:09.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:40:09.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary5 2024-09-16T13:40:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd4 test-primary5 2024-09-16T13:40:09.623 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:40:09.626 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary5 2024-09-16T13:40:09.638 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:40:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary5 2024-09-16T13:40:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:40:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:40:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:40:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:40:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:40:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:40:09.639 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary6 2024-09-16T13:40:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary6 2024-09-16T13:40:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary6 snapshot 10G 2024-09-16T13:40:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary5 2024-09-16T13:40:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:40:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:40:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:40:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:09.640 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary6 10G 2024-09-16T13:40:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:40:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:40:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:40:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:40:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary6 2024-09-16T13:40:09.641 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary5 2024-09-16T13:40:09.642 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary5 2024-09-16T13:40:09.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:40:09.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary5/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary5' 2024-09-16T13:40:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:40:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary6 snapshot 2024-09-16T13:40:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:40:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:40:09.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary6 snapshot 2024-09-16T13:40:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:40:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:40:09.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:40:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:40:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:40:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:40:10.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5961 2024-09-16T13:40:10.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:40:10.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary6 2024-09-16T13:40:11.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary6': 2024-09-16T13:40:11.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:40:11.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:40:11.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:40:11.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 3a7d229a8460 2024-09-16T13:40:11.063 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.3a7d229a8460 2024-09-16T13:40:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:40:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:40:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:40:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:40:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:40:09 2024 2024-09-16T13:40:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:40:09 2024 2024-09-16T13:40:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:40:09 2024 2024-09-16T13:40:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:40:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:40:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 9853a33d-aadd-415b-a589-62bc6312de0f 2024-09-16T13:40:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:40:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:40:11.070 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary6 2024-09-16T13:40:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd5 2024-09-16T13:40:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:40:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd5 2024-09-16T13:40:11.245 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:40:11.348 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:40:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:40:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 1f5d15d6-2508-4436-9c25-72cb7189b877 2024-09-16T13:40:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:40:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:40:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:40:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:40:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:40:11.679 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:40:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:40:11.781 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:40:11.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:40:11.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:40:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:40:11.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:40:11.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 Snapshot ID: 5963 2024-09-16T13:40:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T13:40:11.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:40:11.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary6 2024-09-16T13:40:11.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd5 test-primary6 2024-09-16T13:40:12.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:40:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary6 2024-09-16T13:40:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:40:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary6 2024-09-16T13:40:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:40:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:40:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary7 2024-09-16T13:40:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:40:12.019 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary7 2024-09-16T13:40:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary7 snapshot 10G 2024-09-16T13:40:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary6 2024-09-16T13:40:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:12.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:40:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:40:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:40:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:40:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:40:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:40:12.022 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary6 2024-09-16T13:40:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary7 10G 2024-09-16T13:40:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary6 2024-09-16T13:40:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:40:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:40:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:40:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:40:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:12.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary7 2024-09-16T13:40:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:40:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary6/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary6' 2024-09-16T13:40:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:40:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary7 snapshot 2024-09-16T13:40:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:12.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:40:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:40:12.201 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary7 snapshot 2024-09-16T13:40:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5964 2024-09-16T13:40:12.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:12.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5969 2024-09-16T13:40:12.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:40:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary7 2024-09-16T13:40:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary7': 2024-09-16T13:40:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:40:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:40:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:40:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 3a95a428f3ca 2024-09-16T13:40:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.3a95a428f3ca 2024-09-16T13:40:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:40:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:40:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:40:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:40:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:40:12 2024 2024-09-16T13:40:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:40:12 2024 2024-09-16T13:40:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:40:12 2024 2024-09-16T13:40:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:40:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:40:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 6a9ca88b-8209-4d10-ad07-334e9e0a37da 2024-09-16T13:40:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:40:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:40:13.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary7 2024-09-16T13:40:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd6 2024-09-16T13:40:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:40:13.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd6 2024-09-16T13:40:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:40:13.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:40:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:40:13.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 3d62bb91-aa2c-49bf-8a11-db395ce5ea8c 2024-09-16T13:40:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:40:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:40:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:40:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:40:13.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:40:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): Snapshot ID: 5967 2024-09-16T13:40:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:40:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:40:13.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:40:13.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:13.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T13:40:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:40:14.137 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:40:14.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary7 2024-09-16T13:40:14.140 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd6 test-primary7 2024-09-16T13:40:14.178 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5968 2024-09-16T13:40:14.218 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:14.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:40:14.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary7 2024-09-16T13:40:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:40:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary7 2024-09-16T13:40:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:40:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:40:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:40:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary8 2024-09-16T13:40:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary7 2024-09-16T13:40:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary8 2024-09-16T13:40:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary8 snapshot 10G 2024-09-16T13:40:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:40:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:40:14.351 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:40:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:40:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary7 2024-09-16T13:40:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:40:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary7 2024-09-16T13:40:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:40:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary8 10G 2024-09-16T13:40:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:40:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:40:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:40:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:40:14.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:14.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary8 2024-09-16T13:40:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:40:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary8 snapshot 2024-09-16T13:40:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:40:14.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:40:14.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary8 snapshot 2024-09-16T13:40:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:40:14.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary7/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary7' 2024-09-16T13:40:14.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5974 2024-09-16T13:40:14.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:40:14.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:40:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:40:14.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:40:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary8 2024-09-16T13:40:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary8': 2024-09-16T13:40:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:40:14.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:40:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:40:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 3aa8774dd6cd 2024-09-16T13:40:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.3aa8774dd6cd 2024-09-16T13:40:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:40:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:40:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:40:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:40:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:40:14 2024 2024-09-16T13:40:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:40:14 2024 2024-09-16T13:40:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:40:14 2024 2024-09-16T13:40:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:40:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:40:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: b1662ec3-461a-4192-a479-02fc73ad41c0 2024-09-16T13:40:14.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:40:14.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:40:14.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary8 2024-09-16T13:40:15.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd7 2024-09-16T13:40:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:40:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd7 2024-09-16T13:40:15.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:40:15.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:40:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:40:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: d2dde50c-1960-4c30-a9a4-b41771765798 2024-09-16T13:40:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:40:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:40:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:40:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:40:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:40:15.522 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:40:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:40:15.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:40:15.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary8 2024-09-16T13:40:15.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd7 test-primary8 2024-09-16T13:40:15.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:40:15.697 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary8 2024-09-16T13:40:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:40:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary8 2024-09-16T13:40:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:40:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:40:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:40:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:40:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:40:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:40:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary9 2024-09-16T13:40:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary9 2024-09-16T13:40:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary9 snapshot 10G 2024-09-16T13:40:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:40:15.710 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:40:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:40:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary9 10G 2024-09-16T13:40:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:15.712 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:40:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:40:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:40:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:40:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary9 2024-09-16T13:40:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary8 2024-09-16T13:40:15.713 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary8 2024-09-16T13:40:15.714 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary8 2024-09-16T13:40:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:40:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary9 snapshot 2024-09-16T13:40:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:40:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:40:15.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary9 snapshot 2024-09-16T13:40:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:40:15.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary8/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary8' 2024-09-16T13:40:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5973 2024-09-16T13:40:15.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:15.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:40:15.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:15.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:15.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:40:15.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:40:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:15.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:40:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:40:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:40:16.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5978 2024-09-16T13:40:16.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:16.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5979 2024-09-16T13:40:16.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:40:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:40:16.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:40:16.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:16.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:40:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary9 2024-09-16T13:40:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary9': 2024-09-16T13:40:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:40:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:40:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:40:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 3ab9eb3bb30f 2024-09-16T13:40:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.3ab9eb3bb30f 2024-09-16T13:40:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:40:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:40:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:40:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:40:17.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:40:15 2024 2024-09-16T13:40:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:40:15 2024 2024-09-16T13:40:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:40:15 2024 2024-09-16T13:40:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:40:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:40:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 7cc84c68-2260-4571-ae7c-be71801ff8ef 2024-09-16T13:40:17.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:40:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:40:17.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary9 2024-09-16T13:40:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd8 2024-09-16T13:40:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:40:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd8 2024-09-16T13:40:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:40:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:40:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:40:17.220 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:40:17.293 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:40:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:40:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 91f62c16-12f4-4dc5-83c4-81dc31e4234e 2024-09-16T13:40:17.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:40:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:40:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:40:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:40:17.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:40:17.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:40:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 done 2024-09-16T13:40:17.680 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:40:17.682 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary9 2024-09-16T13:40:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd8 test-primary9 2024-09-16T13:40:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:40:17.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:40:17.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:40:17.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:40:17.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5984 2024-09-16T13:40:17.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5983 2024-09-16T13:40:17.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary9 2024-09-16T13:40:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:40:17.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary9 2024-09-16T13:40:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary9 2024-09-16T13:40:17.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary9 2024-09-16T13:40:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:40:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:40:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary10 2024-09-16T13:40:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ MNTPT=test-primary10 2024-09-16T13:40:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image_and_enable_mirror cluster1 mirror image-primary10 snapshot 10G 2024-09-16T13:40:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:17.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:40:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:40:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n snapshot ']' 2024-09-16T13:40:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:17.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ create_image cluster1 mirror image-primary10 10G 2024-09-16T13:40:17.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:17.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:17.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:17.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:17.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:40:17.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:17.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local size=128 2024-09-16T13:40:17.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 10G ']' 2024-09-16T13:40:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ size=10G 2024-09-16T13:40:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ shift 2024-09-16T13:40:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 create --size 10G --image-feature layering,exclusive-lock,object-map,fast-diff mirror/image-primary10 2024-09-16T13:40:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary9 2024-09-16T13:40:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:40:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:40:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:40:17.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:40:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:40:18.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary9/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary9' 2024-09-16T13:40:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' image = image ']' 2024-09-16T13:40:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ enable_mirror cluster1 mirror image-primary10 snapshot 2024-09-16T13:40:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:40:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mode=snapshot 2024-09-16T13:40:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image enable mirror/image-primary10 snapshot 2024-09-16T13:40:18.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5989 2024-09-16T13:40:18.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5988 2024-09-16T13:40:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:18.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:18.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:40:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:40:18.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:40:18.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5992 2024-09-16T13:40:18.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Mirroring enabled 2024-09-16T13:40:18.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 info mirror/image-primary10 2024-09-16T13:40:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:rbd image 'image-primary10': 2024-09-16T13:40:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: size 10 GiB in 2560 objects 2024-09-16T13:40:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: order 22 (4 MiB objects) 2024-09-16T13:40:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: snapshot_count: 1 2024-09-16T13:40:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: id: 3ad5b097b0d4 2024-09-16T13:40:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: block_name_prefix: rbd_data.3ad5b097b0d4 2024-09-16T13:40:19.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: format: 2 2024-09-16T13:40:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: features: layering, exclusive-lock, object-map, fast-diff 2024-09-16T13:40:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: op_features: 2024-09-16T13:40:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: flags: 2024-09-16T13:40:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: create_timestamp: Mon Sep 16 13:40:18 2024 2024-09-16T13:40:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: access_timestamp: Mon Sep 16 13:40:18 2024 2024-09-16T13:40:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: modify_timestamp: Mon Sep 16 13:40:18 2024 2024-09-16T13:40:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring state: enabled 2024-09-16T13:40:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring mode: snapshot 2024-09-16T13:40:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring global id: 6a68c4a7-d310-4320-ab9d-39bbe0aa88e6 2024-09-16T13:40:19.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: mirroring primary: true 2024-09-16T13:40:19.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:40:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster1 device map -t nbd -o try-netlink mirror/image-primary10 2024-09-16T13:40:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEV=/dev/nbd9 2024-09-16T13:40:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ DEVS+=($DEV) 2024-09-16T13:40:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mkfs.ext4 /dev/nbd9 2024-09-16T13:40:19.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:mke2fs 1.45.5 (07-Jan-2020) 2024-09-16T13:40:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Discarding device blocks: 4096/2621440 done 2024-09-16T13:40:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating filesystem with 2621440 4k blocks and 655360 inodes 2024-09-16T13:40:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Filesystem UUID: 3a40089a-86cb-4a3a-a5bf-0694b2b26a70 2024-09-16T13:40:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Superblock backups stored on blocks: 2024-09-16T13:40:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 2024-09-16T13:40:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:40:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Allocating group tables: 0/80 done 2024-09-16T13:40:19.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing inode tables: 0/80 done 2024-09-16T13:40:19.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Creating journal (16384 blocks): done 2024-09-16T13:40:19.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:19.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:40:19.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:19.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:19.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:40:19.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:40:19.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Writing superblocks and filesystem accounting information: 0/80 Snapshot ID: 5990 2024-09-16T13:40:19.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:40:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:40:19.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:40:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:done 2024-09-16T13:40:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: 2024-09-16T13:40:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mkdir test-primary10 2024-09-16T13:40:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo mount /dev/nbd9 test-primary10 2024-09-16T13:40:20.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ whoami 2024-09-16T13:40:20.227 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo chown ubuntu test-primary10 2024-09-16T13:40:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ SNAP_PIDS+=($!) 2024-09-16T13:40:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ take_mirror_snapshots cluster1 mirror image-primary10 2024-09-16T13:40:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:40:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:40:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ WORKLOAD_PIDS+=($!) 2024-09-16T13:40:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:40:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:40:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ slow_untar_workload test-primary10 2024-09-16T13:40:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:40:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 132386 2024-09-16T13:40:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mountpt=test-primary10 2024-09-16T13:40:20.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cp linux-5.4.tar.gz test-primary10 2024-09-16T13:40:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local ret=0 2024-09-16T13:40:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ timeout 5m bash -c 'zcat test-primary10/linux-5.4.tar.gz | pv -L 256K | tar xf - -C test-primary10' 2024-09-16T13:40:20.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5996 2024-09-16T13:40:20.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:40:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:40:20.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:40:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:40:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:40:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:40:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6000 2024-09-16T13:40:21.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:21.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:21.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:40:21.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:21.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:21.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:40:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:40:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:40:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:40:21.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:40:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5998 2024-09-16T13:40:21.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 5997 2024-09-16T13:40:21.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:21.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:40:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:40:21.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:40:22.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6004 2024-09-16T13:40:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6003 2024-09-16T13:40:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:22.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:40:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:40:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:40:23.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6005 2024-09-16T13:40:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:40:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:40:23.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:40:23.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6009 2024-09-16T13:40:23.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:23.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6008 2024-09-16T13:40:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:40:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:40:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:40:24.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6012 2024-09-16T13:40:24.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:40:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:40:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:40:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:40:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:40:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:40:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:40:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:40:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:40:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:40:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:40:25.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:40:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6014 2024-09-16T13:40:25.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6016 2024-09-16T13:40:25.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:25.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:40:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:40:26.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:40:26.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6019 2024-09-16T13:40:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6020 2024-09-16T13:40:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:40:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:40:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:40:26.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:26.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:40:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:40:27.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:40:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:40:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:40:27.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:40:27.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6023 2024-09-16T13:40:27.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6022 2024-09-16T13:40:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:28.739 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6028 2024-09-16T13:40:28.759 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6027 2024-09-16T13:40:28.763 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6029 2024-09-16T13:40:28.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:28.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:40:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:40:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:40:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:40:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:40:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:40:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:40:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:40:29.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:40:29.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6031 2024-09-16T13:40:29.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:40:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:40:29.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:40:30.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6036 2024-09-16T13:40:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:30.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6035 2024-09-16T13:40:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:40:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:40:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:40:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:40:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:40:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:40:30.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:40:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:31.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:40:31.766 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:40:31.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:31.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:40:31.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:31.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:31.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:40:31.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:40:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:40:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:40:31.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:40:31.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6039 2024-09-16T13:40:31.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6040 2024-09-16T13:40:31.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:32.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6044 2024-09-16T13:40:32.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6043 2024-09-16T13:40:32.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6045 2024-09-16T13:40:32.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:40:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:40:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:40:32.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:33.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:33.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6047 2024-09-16T13:40:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:40:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:40:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:40:33.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6046 2024-09-16T13:40:33.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:34.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:34.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:40:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:40:34.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:40:34.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6053 2024-09-16T13:40:34.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:34.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:40:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:40:34.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:40:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:40:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:40:35.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:40:35.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6056 2024-09-16T13:40:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:40:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:40:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:40:36.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6055 2024-09-16T13:40:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:40:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:40:36.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:40:36.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:40:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:40:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:40:36.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6060 2024-09-16T13:40:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6059 2024-09-16T13:40:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:40:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:40:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:40:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:40:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:40:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:40:37.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:37.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6062 2024-09-16T13:40:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:40:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:40:37.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:40:37.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6063 2024-09-16T13:40:37.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6064 2024-09-16T13:40:37.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:38.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6068 2024-09-16T13:40:38.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6067 2024-09-16T13:40:38.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:38.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:40:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:40:39.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:40:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:40:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:39.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:40:39.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:40:39.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6071 2024-09-16T13:40:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:40:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:40:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:40:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:40:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:40:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:40:40.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6075 2024-09-16T13:40:40.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6076 2024-09-16T13:40:40.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:40.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:40:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:40:40.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:40:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:40:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:40:41.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:40:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:40:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:40:41.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:40:41.779 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6079 2024-09-16T13:40:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:40:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:41.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:40:41.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:40:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:40:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:40:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:40:41.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6080 2024-09-16T13:40:41.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:41.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6084 2024-09-16T13:40:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:40:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:40:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:40:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6083 2024-09-16T13:40:42.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6085 2024-09-16T13:40:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:42.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:43.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6087 2024-09-16T13:40:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:40:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:40:43.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:40:43.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6086 2024-09-16T13:40:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:40:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:40:43.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:40:43.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:43.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:44.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6093 2024-09-16T13:40:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:40:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:40:44.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:40:44.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:40:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:40:44.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:40:45.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6095 2024-09-16T13:40:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:40:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:40:45.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:40:45.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:45.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6096 2024-09-16T13:40:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:40:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:40:45.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:40:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:40:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:40:45.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:40:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:46.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6100 2024-09-16T13:40:46.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6098 2024-09-16T13:40:46.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:46.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:40:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:40:46.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:40:46.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:46.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:40:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:40:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:40:47.793 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6102 2024-09-16T13:40:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:40:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:40:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:40:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6104 2024-09-16T13:40:47.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6105 2024-09-16T13:40:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6106 2024-09-16T13:40:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6108 2024-09-16T13:40:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:48.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:40:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:40:48.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:40:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:40:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:40:48.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:40:49.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6111 2024-09-16T13:40:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:40:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:40:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:40:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:40:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:49.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:40:49.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:40:49.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:40:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:40:50.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:40:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6115 2024-09-16T13:40:50.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6116 2024-09-16T13:40:50.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:40:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:40:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:40:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:40:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:40:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:40:50.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:40:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:40:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:40:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:40:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:40:51.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:40:51.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6119 2024-09-16T13:40:51.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6118 2024-09-16T13:40:51.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:51.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:40:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:40:52.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:40:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6121 2024-09-16T13:40:52.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6125 2024-09-16T13:40:52.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6124 2024-09-16T13:40:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:52.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:40:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:40:53.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:40:53.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6127 2024-09-16T13:40:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:40:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:40:53.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:40:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6126 2024-09-16T13:40:53.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:53.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:54.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6131 2024-09-16T13:40:54.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:40:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:40:54.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:40:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:40:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:40:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:40:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:40:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:40:55.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:40:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:40:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:40:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:40:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:40:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:55.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:40:55.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:40:55.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6137 2024-09-16T13:40:55.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6135 2024-09-16T13:40:55.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:55.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:56.854 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:40:56.845+0000 7f61b1855700 -1 reset not still connected to 0x557b1f47f930 2024-09-16T13:40:56.855 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:40:56.845+0000 7f61b1855700 -1 reset not still connected to 0x557b1f47fd40 2024-09-16T13:40:56.862 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:40:56.853+0000 7f61b1855700 -1 reset not still connected to 0x557b229664e0 2024-09-16T13:40:56.862 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:40:56.853+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T13:40:56.863 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:40:56.853+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T13:40:56.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6141 2024-09-16T13:40:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6140 2024-09-16T13:40:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:40:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:40:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:40:56.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:40:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:40:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:40:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:40:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:40:57.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:40:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6143 2024-09-16T13:40:57.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6142 2024-09-16T13:40:57.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6144 2024-09-16T13:40:57.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:58.761 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6148 2024-09-16T13:40:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6147 2024-09-16T13:40:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:58.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:40:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:40:58.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:40:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:40:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:40:58.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:40:59.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6152 2024-09-16T13:40:59.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:40:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:40:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:59.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:40:59.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:40:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:40:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:40:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:40:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:40:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:40:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:41:00.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6157 2024-09-16T13:41:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6156 2024-09-16T13:41:00.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:00.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:41:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:41:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:41:00.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:41:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:41:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:41:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:41:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:41:00.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:41:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:41:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:41:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:41:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:41:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:41:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:41:01.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6161 2024-09-16T13:41:01.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:01.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6160 2024-09-16T13:41:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:02.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:41:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:41:02.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:41:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6165 2024-09-16T13:41:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6164 2024-09-16T13:41:02.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6166 2024-09-16T13:41:02.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:02.942 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:41:03.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:41:03.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:41:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:41:03.896 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:41:03.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:41:03.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6168 2024-09-16T13:41:03.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6167 2024-09-16T13:41:03.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:03.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:04.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:41:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:41:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:41:04.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6172 2024-09-16T13:41:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:04.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:41:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:41:04.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:41:04.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:05.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:41:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:41:05.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:41:05.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6175 2024-09-16T13:41:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:41:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:41:05.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:41:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:41:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:41:05.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:41:05.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6176 2024-09-16T13:41:05.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:05.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:06.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6181 2024-09-16T13:41:06.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:41:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:41:06.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:41:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:41:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:41:06.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:41:06.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6179 2024-09-16T13:41:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6186 2024-09-16T13:41:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:41:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:41:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:41:07.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6184 2024-09-16T13:41:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6183 2024-09-16T13:41:07.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:07.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:08.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6188 2024-09-16T13:41:08.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6191 2024-09-16T13:41:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:41:08.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:41:08.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:41:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:41:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:41:08.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:41:09.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:41:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:41:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:41:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6194 2024-09-16T13:41:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:09.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:41:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:41:09.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:41:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:41:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:41:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:41:10.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6197 2024-09-16T13:41:10.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6196 2024-09-16T13:41:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:41:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:41:10.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:41:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:41:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:41:10.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:41:10.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:41:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:41:11.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:41:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:41:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:41:11.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:41:11.940 INFO:tasks.ceph.cluster1.osd.1.smithi018.stderr:2024-09-16T13:41:11.933+0000 7f4071c2f700 -1 reset not still connected to 0x563591d7dad0 2024-09-16T13:41:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6198 2024-09-16T13:41:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:11.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6201 2024-09-16T13:41:11.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:41:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:41:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:41:12.960 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:41:12.949+0000 7f61b1855700 -1 reset not still connected to 0x557b3b6935f0 2024-09-16T13:41:12.967 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:41:12.957+0000 7f61b1855700 -1 reset not still connected to 0x557b3b7a4410 2024-09-16T13:41:12.967 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:41:12.957+0000 7f61b1855700 -1 reset not still connected to 0x557b3f505790 2024-09-16T13:41:12.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6205 2024-09-16T13:41:12.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6204 2024-09-16T13:41:12.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:12.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:13.010 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6206 2024-09-16T13:41:13.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:41:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:13.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:13.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:41:13.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:41:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6207 2024-09-16T13:41:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:41:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:41:13.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:41:13.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6208 2024-09-16T13:41:13.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:13.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:14.955 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6213 2024-09-16T13:41:14.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:41:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:41:14.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:41:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:41:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:41:14.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:41:15.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6217 2024-09-16T13:41:15.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6216 2024-09-16T13:41:15.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:41:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:41:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:41:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:41:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:41:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:41:15.998 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:41:16.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:41:16.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:41:16.732 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:41:16.725+0000 7ff4f6946700 -1 reset not still connected to 0x5575793161a0 2024-09-16T13:41:16.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6220 2024-09-16T13:41:16.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6221 2024-09-16T13:41:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:41:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:41:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:41:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:41:16.990 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:41:16.991 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:41:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:17.003 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:17.656 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:41:17.643+0000 7f74def64700 -1 reset not still connected to 0x55943f382b60 2024-09-16T13:41:17.659 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:41:17.647+0000 7f74def64700 -1 reset not still connected to 0x559441c00750 2024-09-16T13:41:17.660 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:41:17.647+0000 7f74def64700 -1 reset not still connected to 0x559441c00a90 2024-09-16T13:41:17.660 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:41:17.647+0000 7f74def64700 -1 reset not still connected to 0x559458e715f0 2024-09-16T13:41:17.661 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:41:17.651+0000 7f74def64700 -1 reset not still connected to 0x559459066dd0 2024-09-16T13:41:17.661 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:41:17.651+0000 7f74def64700 -1 reset not still connected to 0x5594648c7930 2024-09-16T13:41:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:41:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:41:17.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:41:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6226 2024-09-16T13:41:18.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6225 2024-09-16T13:41:18.020 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6224 2024-09-16T13:41:18.021 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:18.030 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:18.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6231 2024-09-16T13:41:18.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6228 2024-09-16T13:41:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:41:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:41:18.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:41:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:41:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:41:19.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:41:19.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6232 2024-09-16T13:41:19.798 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:41:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:41:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:41:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:41:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:41:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:41:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6236 2024-09-16T13:41:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6237 2024-09-16T13:41:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:20.877 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:41:20.859+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8bf12bad0 2024-09-16T13:41:20.878 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:41:20.859+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c299a4e0 2024-09-16T13:41:20.878 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:41:20.859+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c299aa90 2024-09-16T13:41:20.878 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:41:20.859+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c299bd40 2024-09-16T13:41:20.878 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:41:20.859+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c8c76ea0 2024-09-16T13:41:20.878 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:41:20.867+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8de10aa90 2024-09-16T13:41:20.878 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:41:20.867+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8e1c1edd0 2024-09-16T13:41:20.878 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:41:20.867+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8e3b55040 2024-09-16T13:41:20.878 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:41:20.867+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8e97e8dd0 2024-09-16T13:41:21.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:41:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:41:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:41:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:41:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:41:21.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:41:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:41:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:41:21.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:41:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6240 2024-09-16T13:41:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6241 2024-09-16T13:41:21.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:21.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:21.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:41:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:41:21.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:41:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:41:21.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:41:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:41:22.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6244 2024-09-16T13:41:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:41:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:41:22.802 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:41:22.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6246 2024-09-16T13:41:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6245 2024-09-16T13:41:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:22.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:23.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6251 2024-09-16T13:41:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6250 2024-09-16T13:41:23.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:41:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:41:23.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:41:23.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:23.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:41:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:41:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:41:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:41:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:41:24.807 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:41:24.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6254 2024-09-16T13:41:24.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:41:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:41:24.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:41:24.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:25.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6257 2024-09-16T13:41:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:41:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:41:25.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:41:25.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:25.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:41:25.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:25.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:25.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:41:25.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:41:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:41:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:41:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:41:25.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6256 2024-09-16T13:41:25.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:41:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:26.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:41:26.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:41:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:41:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:26.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:41:26.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:41:26.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6260 2024-09-16T13:41:26.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:26.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6261 2024-09-16T13:41:26.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:41:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:41:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:41:27.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6266 2024-09-16T13:41:27.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6263 2024-09-16T13:41:27.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6265 2024-09-16T13:41:27.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:27.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:27.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6267 2024-09-16T13:41:28.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6268 2024-09-16T13:41:28.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:41:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:41:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:41:28.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:41:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:41:28.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:41:29.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6272 2024-09-16T13:41:29.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:41:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:41:29.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:41:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:41:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:41:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:41:30.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6277 2024-09-16T13:41:30.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6276 2024-09-16T13:41:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:41:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:41:30.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:41:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:30.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:30.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:41:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:41:30.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:41:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:41:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:41:30.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:41:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:31.805 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:41:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:41:31.806 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:41:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:41:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:41:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:41:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6281 2024-09-16T13:41:31.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6280 2024-09-16T13:41:31.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:41:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:41:32.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:41:32.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6284 2024-09-16T13:41:32.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6285 2024-09-16T13:41:32.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6286 2024-09-16T13:41:32.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:32.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:41:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:41:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:41:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:41:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:41:33.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:41:33.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6288 2024-09-16T13:41:33.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6287 2024-09-16T13:41:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:41:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:41:34.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:41:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:41:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:34.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:41:34.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:41:34.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6292 2024-09-16T13:41:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:35.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6296 2024-09-16T13:41:35.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6297 2024-09-16T13:41:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:41:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:41:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:41:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:41:35.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:41:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:41:35.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:41:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:41:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:41:35.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6300 2024-09-16T13:41:36.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:41:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:41:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:41:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:41:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:41:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:41:36.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:36.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6301 2024-09-16T13:41:36.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:41:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:41:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:41:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6304 2024-09-16T13:41:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6306 2024-09-16T13:41:37.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6305 2024-09-16T13:41:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:37.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:37.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:38.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6311 2024-09-16T13:41:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:38.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6310 2024-09-16T13:41:38.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:41:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:41:38.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:41:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:41:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:41:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:41:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6312 2024-09-16T13:41:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:41:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:39.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:41:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:41:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:41:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:41:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:41:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6317 2024-09-16T13:41:40.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6316 2024-09-16T13:41:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:40.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:41:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:41:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:41:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:41:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:41:40.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:41:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:41:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:41:40.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:41:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:41:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:41:41.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:41:41.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:41:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:41:41.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:41:41.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6321 2024-09-16T13:41:41.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6320 2024-09-16T13:41:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:42.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:42.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:41:42.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:42.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:42.904 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:41:42.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:41:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6325 2024-09-16T13:41:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6324 2024-09-16T13:41:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6326 2024-09-16T13:41:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:42.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:43.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6327 2024-09-16T13:41:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:41:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:41:43.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:41:43.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:43.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:41:43.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:43.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:43.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:41:43.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:41:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6328 2024-09-16T13:41:43.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:43.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6334 2024-09-16T13:41:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:41:44.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:41:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:41:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:41:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:41:44.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:41:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6336 2024-09-16T13:41:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6337 2024-09-16T13:41:45.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:45.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:45.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:41:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:41:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:41:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:41:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:41:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:41:45.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:41:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:41:45.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:41:46.793 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:41:46.783+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c299bd40 2024-09-16T13:41:46.797 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:41:46.783+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c8c76ea0 2024-09-16T13:41:46.797 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:41:46.783+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8de10aa90 2024-09-16T13:41:46.797 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:41:46.783+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8e1c1edd0 2024-09-16T13:41:46.797 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:41:46.783+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8e3b55040 2024-09-16T13:41:46.798 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:41:46.783+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8e97e8dd0 2024-09-16T13:41:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:41:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:46.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:41:46.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:41:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:41:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:41:46.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:41:46.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6341 2024-09-16T13:41:46.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6340 2024-09-16T13:41:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:41:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:41:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:41:47.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6345 2024-09-16T13:41:47.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6344 2024-09-16T13:41:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6346 2024-09-16T13:41:47.994 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:41:47.979+0000 7f74def64700 -1 reset not still connected to 0x55943e1b5ba0 2024-09-16T13:41:47.994 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:41:47.979+0000 7f74def64700 -1 reset not still connected to 0x55943f382b60 2024-09-16T13:41:47.994 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:41:47.979+0000 7f74def64700 -1 reset not still connected to 0x559441c00750 2024-09-16T13:41:47.994 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:41:47.979+0000 7f74def64700 -1 reset not still connected to 0x559441c00a90 2024-09-16T13:41:47.994 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:41:47.979+0000 7f74def64700 -1 reset not still connected to 0x559458e715f0 2024-09-16T13:41:47.994 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:41:47.979+0000 7f74def64700 -1 reset not still connected to 0x559459066dd0 2024-09-16T13:41:47.994 INFO:tasks.ceph.cluster2.osd.0.smithi172.stderr:2024-09-16T13:41:47.979+0000 7f74def64700 -1 reset not still connected to 0x5594648c7930 2024-09-16T13:41:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:48.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:48.006 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6351 2024-09-16T13:41:48.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:48.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6347 2024-09-16T13:41:48.824 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:48.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:41:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:41:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:41:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:41:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:41:48.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:41:49.792 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6352 2024-09-16T13:41:49.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:41:49.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:41:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:41:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:41:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:41:49.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:41:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6357 2024-09-16T13:41:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6356 2024-09-16T13:41:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:50.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:41:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:41:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:41:51.008 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:41:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:41:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:41:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:41:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:41:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:41:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6360 2024-09-16T13:41:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:41:51.813 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:41:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:41:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6361 2024-09-16T13:41:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:51.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:41:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:41:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:41:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:52.809 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:41:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:41:52.810 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:41:52.811 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6364 2024-09-16T13:41:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6366 2024-09-16T13:41:52.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6365 2024-09-16T13:41:52.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:52.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:53.804 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6371 2024-09-16T13:41:53.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6370 2024-09-16T13:41:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:41:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:41:53.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:41:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:41:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:41:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:41:53.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:54.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:54.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:41:54.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:54.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:54.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:41:54.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:41:54.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6374 2024-09-16T13:41:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:41:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:54.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:41:54.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:41:54.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6376 2024-09-16T13:41:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:41:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:41:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:41:55.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6377 2024-09-16T13:41:55.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:41:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:41:55.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:41:55.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:41:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:41:55.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:41:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:41:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:41:56.830 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:41:56.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6380 2024-09-16T13:41:56.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6381 2024-09-16T13:41:56.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:41:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:41:56.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:41:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:41:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:57.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:41:57.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:41:57.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6386 2024-09-16T13:41:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6385 2024-09-16T13:41:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6384 2024-09-16T13:41:57.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:57.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6387 2024-09-16T13:41:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6388 2024-09-16T13:41:58.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:58.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:41:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:41:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:41:58.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:41:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:41:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:41:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:41:59.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:59.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:41:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:41:59.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:41:59.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:41:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:41:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:41:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:41:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:41:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:41:59.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6393 2024-09-16T13:41:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:00.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6397 2024-09-16T13:42:00.858 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6396 2024-09-16T13:42:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:42:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:00.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:42:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:42:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:42:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:42:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:42:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:00.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:42:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:42:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:42:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:42:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:42:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:42:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:42:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:42:01.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:42:01.882 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6398 2024-09-16T13:42:01.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6400 2024-09-16T13:42:01.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:01.916 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:02.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:42:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:42:02.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:42:02.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6405 2024-09-16T13:42:02.893 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6406 2024-09-16T13:42:02.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6404 2024-09-16T13:42:02.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:02.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:02.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:42:03.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:42:03.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:42:03.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:03.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:42:03.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:03.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:03.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:42:03.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:42:03.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6408 2024-09-16T13:42:03.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6407 2024-09-16T13:42:03.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:04.884 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6414 2024-09-16T13:42:04.900 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:42:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:42:04.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:42:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:42:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:42:04.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:42:05.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6417 2024-09-16T13:42:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6415 2024-09-16T13:42:05.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:42:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:42:05.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:42:05.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:42:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:05.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:42:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:42:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:42:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:42:05.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:42:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:42:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:42:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:42:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:42:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:06.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:42:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:42:06.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6421 2024-09-16T13:42:06.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6420 2024-09-16T13:42:06.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:06.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:06.946 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:42:06.936+0000 7f61b1855700 -1 reset not still connected to 0x557b3a126820 2024-09-16T13:42:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:42:07.902 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:42:07.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:42:07.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6426 2024-09-16T13:42:07.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6424 2024-09-16T13:42:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6425 2024-09-16T13:42:07.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:07.945 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:07.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6429 2024-09-16T13:42:08.833 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6431 2024-09-16T13:42:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:08.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:08.910 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:42:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:42:08.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:42:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:42:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:42:08.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:42:09.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6432 2024-09-16T13:42:09.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:42:09.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:42:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:42:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:42:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:42:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:42:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6437 2024-09-16T13:42:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6436 2024-09-16T13:42:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:10.930 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:42:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:42:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:42:10.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:42:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:42:10.950 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:42:10.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:10.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:42:10.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:10.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:10.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:42:10.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:42:11.833 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:42:11.819+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8e97e8dd0 2024-09-16T13:42:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:42:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:42:11.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:42:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:42:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:42:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:42:11.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6440 2024-09-16T13:42:11.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6441 2024-09-16T13:42:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:11.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:42:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:42:12.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:42:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6443 2024-09-16T13:42:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6445 2024-09-16T13:42:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6444 2024-09-16T13:42:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:42:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:42:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:42:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:13.932 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:42:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:42:13.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:42:13.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6447 2024-09-16T13:42:13.947 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6448 2024-09-16T13:42:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:13.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6452 2024-09-16T13:42:14.946 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:42:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:42:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:42:14.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:14.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:42:14.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:14.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:14.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:42:14.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:42:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6457 2024-09-16T13:42:15.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6456 2024-09-16T13:42:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:42:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:42:15.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:42:15.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:15.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:42:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:42:15.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:42:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:42:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:42:15.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:42:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6460 2024-09-16T13:42:16.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:42:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:42:16.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:42:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:42:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:42:16.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:42:16.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6461 2024-09-16T13:42:16.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:16.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:42:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:42:17.948 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:42:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6464 2024-09-16T13:42:17.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6465 2024-09-16T13:42:17.997 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6466 2024-09-16T13:42:18.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:18.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:18.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:18.820 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6470 2024-09-16T13:42:18.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6471 2024-09-16T13:42:18.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:18.844 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:18.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:42:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:42:18.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:42:18.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:18.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:42:18.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:18.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:18.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:42:18.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:42:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6472 2024-09-16T13:42:19.825 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:19.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:42:19.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:19.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:19.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:42:19.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:42:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:42:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:42:19.980 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:42:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6476 2024-09-16T13:42:20.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6477 2024-09-16T13:42:20.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:42:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:42:21.005 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:42:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:42:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:42:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:42:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:42:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:42:21.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:42:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6480 2024-09-16T13:42:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6481 2024-09-16T13:42:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:21.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:21.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:42:21.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:21.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:21.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:42:21.846 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:42:21.847 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:42:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:42:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:42:21.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:42:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:22.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:42:22.832 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:42:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6485 2024-09-16T13:42:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6486 2024-09-16T13:42:22.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6484 2024-09-16T13:42:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:22.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:22.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:23.812 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6490 2024-09-16T13:42:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:42:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:42:23.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:42:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:42:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:42:23.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:42:23.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6491 2024-09-16T13:42:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6494 2024-09-16T13:42:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:42:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:24.849 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:42:24.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:42:24.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:24.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:42:24.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:24.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:24.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:42:24.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:42:24.871 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:42:24.847+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c299a4e0 2024-09-16T13:42:24.871 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:42:24.847+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c299aa90 2024-09-16T13:42:24.871 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:42:24.851+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c299bd40 2024-09-16T13:42:24.871 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:42:24.851+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c8c76ea0 2024-09-16T13:42:24.871 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:42:24.851+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8de10aa90 2024-09-16T13:42:24.871 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:42:24.851+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8e1c1edd0 2024-09-16T13:42:24.872 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:42:24.851+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8e3b55040 2024-09-16T13:42:24.872 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:42:24.851+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8e97e8dd0 2024-09-16T13:42:25.839 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6496 2024-09-16T13:42:25.850 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6497 2024-09-16T13:42:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:25.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:25.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:42:25.870 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:42:25.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:42:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary1 2024-09-16T13:42:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:42:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary1 2024-09-16T13:42:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:42:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:42:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:42:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary2 2024-09-16T13:42:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:42:26.843 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary2 2024-09-16T13:42:26.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6500 2024-09-16T13:42:26.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6501 2024-09-16T13:42:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:42:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:42:26.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:42:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:27.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:42:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:42:27.841 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:42:27.842 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:42:27.832+0000 7f61b1855700 -1 reset not still connected to 0x557b3b7a4410 2024-09-16T13:42:27.842 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:42:27.832+0000 7f61b1855700 -1 reset not still connected to 0x557b3f505790 2024-09-16T13:42:27.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6504 2024-09-16T13:42:27.898 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6506 2024-09-16T13:42:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:27.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6505 2024-09-16T13:42:27.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6508 2024-09-16T13:42:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6511 2024-09-16T13:42:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:42:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:42:28.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:42:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:42:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:42:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:42:29.851 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6512 2024-09-16T13:42:29.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary3 2024-09-16T13:42:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:42:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary3 2024-09-16T13:42:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:42:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:42:29.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:42:30.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6517 2024-09-16T13:42:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6516 2024-09-16T13:42:30.885 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:42:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:42:30.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:42:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:42:30.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 132524 2024-09-16T13:42:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:42:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:42:30.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:42:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:42:31.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 132701 2024-09-16T13:42:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary4 2024-09-16T13:42:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:42:31.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary4 2024-09-16T13:42:31.892 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6521 2024-09-16T13:42:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6520 2024-09-16T13:42:31.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:42:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:32.864 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:42:32.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:42:32.886 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6524 2024-09-16T13:42:32.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:32.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6525 2024-09-16T13:42:32.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:42:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:42:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:42:33.889 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6528 2024-09-16T13:42:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary5 2024-09-16T13:42:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:42:33.890 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary5 2024-09-16T13:42:33.903 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:34.894 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6530 2024-09-16T13:42:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:34.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:42:34.912 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 132856 2024-09-16T13:42:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:42:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:42:34.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:42:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary6 2024-09-16T13:42:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:42:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6532 2024-09-16T13:42:35.906 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary6 2024-09-16T13:42:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6533 2024-09-16T13:42:35.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:42:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:35.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:42:35.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:42:35.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:36.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:42:36.905 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 132999 2024-09-16T13:42:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6536 2024-09-16T13:42:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:37.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary7 2024-09-16T13:42:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:42:37.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary7 2024-09-16T13:42:37.921 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6539 2024-09-16T13:42:37.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6538 2024-09-16T13:42:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:38.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:42:38.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 133214 2024-09-16T13:42:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:42:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:42:38.933 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:42:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6542 2024-09-16T13:42:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:39.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary8 2024-09-16T13:42:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:42:39.929 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary8 2024-09-16T13:42:40.908 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6544 2024-09-16T13:42:40.920 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary9 2024-09-16T13:42:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:42:40.938 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary9 2024-09-16T13:42:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:42:40.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 133446 2024-09-16T13:42:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6546 2024-09-16T13:42:41.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:42:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 133601 2024-09-16T13:42:42.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6548 2024-09-16T13:42:42.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for i in {1..30} 2024-09-16T13:42:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ mirror_image_snapshot cluster1 mirror image-primary10 2024-09-16T13:42:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:42:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:42:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:42:43.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 mirror image snapshot mirror/image-primary10 2024-09-16T13:42:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:42:44.944 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 133843 2024-09-16T13:42:45.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${SNAP_PIDS[@]} 2024-09-16T13:42:45.939 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 134080 2024-09-16T13:42:45.943 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Snapshot ID: 6550 2024-09-16T13:42:45.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 3 2024-09-16T13:42:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T13:42:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:42:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 132387 2024-09-16T13:45:01.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:45:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:45:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:45:01.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 132525 2024-09-16T13:45:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:45:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:45:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:45:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 132702 2024-09-16T13:45:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:45:06.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:45:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:45:06.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 132857 2024-09-16T13:45:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:45:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:45:08.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:45:08.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 133000 2024-09-16T13:45:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:45:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:45:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:45:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 133215 2024-09-16T13:45:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:45:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:45:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:45:12.154 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 133447 2024-09-16T13:45:14.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:45:14.477 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:45:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:45:14.478 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 133602 2024-09-16T13:45:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:45:15.834 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:45:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:45:15.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 133845 2024-09-16T13:45:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:45:18.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:45:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${WORKLOAD_PIDS[@]} 2024-09-16T13:45:18.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 134081 2024-09-16T13:45:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ret=124 2024-09-16T13:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( ret != 124 )) 2024-09-16T13:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T13:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 1 2024-09-16T13:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd0 2024-09-16T13:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary1 2024-09-16T13:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary1 2024-09-16T13:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary1 2024-09-16T13:45:20.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 2 2024-09-16T13:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd1 2024-09-16T13:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary2 2024-09-16T13:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary2 2024-09-16T13:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary2 2024-09-16T13:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 3 2024-09-16T13:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd2 2024-09-16T13:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:45:20.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary3 2024-09-16T13:45:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary3 2024-09-16T13:45:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:45:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary3 2024-09-16T13:45:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:45:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 4 2024-09-16T13:45:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:45:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd3 2024-09-16T13:45:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:45:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary4 2024-09-16T13:45:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary4 2024-09-16T13:45:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:45:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary4 2024-09-16T13:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 5 2024-09-16T13:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd4 2024-09-16T13:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary5 2024-09-16T13:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary5 2024-09-16T13:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary5 2024-09-16T13:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 7 2024-09-16T13:45:20.382 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 6 2024-09-16T13:45:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd6 2024-09-16T13:45:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:45:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd5 2024-09-16T13:45:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary6 2024-09-16T13:45:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary6 2024-09-16T13:45:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:45:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:45:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary6 2024-09-16T13:45:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 8 2024-09-16T13:45:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary7 2024-09-16T13:45:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary7 2024-09-16T13:45:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:45:20.383 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary7 2024-09-16T13:45:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:45:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:45:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd7 2024-09-16T13:45:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary8 2024-09-16T13:45:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 9 2024-09-16T13:45:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary8 2024-09-16T13:45:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:45:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary8 2024-09-16T13:45:20.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ COMPARE_PIDS+=($!) 2024-09-16T13:45:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:45:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 139887 2024-09-16T13:45:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd8 2024-09-16T13:45:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary9 2024-09-16T13:45:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary9 2024-09-16T13:45:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:45:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary9 2024-09-16T13:45:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ compare_demoted_promoted_image 10 2024-09-16T13:45:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local dev=/dev/nbd9 2024-09-16T13:45:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local img=image-primary10 2024-09-16T13:45:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local mntpt=test-primary10 2024-09-16T13:45:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local demote_md5 promote_md5 2024-09-16T13:45:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo umount test-primary10 2024-09-16T13:45:20.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:45:20.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd2 2024-09-16T13:45:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd9 2024-09-16T13:45:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:45:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd4 2024-09-16T13:45:20.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:45:20.975 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd0 2024-09-16T13:45:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd6 2024-09-16T13:45:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:45:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:45:20.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd3 2024-09-16T13:45:20.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:45:20.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:45:20.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd1 2024-09-16T13:45:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd5 2024-09-16T13:45:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd8 2024-09-16T13:45:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:45:21.007 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:45:21.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd7 2024-09-16T13:45:21.014 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:46:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=bd2d5ff43de51882724594e95a408320 2024-09-16T13:46:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary7 2024-09-16T13:46:17.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=91a6db6b9eb83b19755c4222aceef3a4 2024-09-16T13:46:17.086 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary2 2024-09-16T13:46:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=ac46edaaae23672000609a93d1cb63d1 2024-09-16T13:46:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary6 2024-09-16T13:46:18.026 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=7c0ebb690230d30433c3b0023b29a388 2024-09-16T13:46:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary10 2024-09-16T13:46:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=a419c17ba3b7aa1e4365fd64e5961b41 2024-09-16T13:46:18.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary5 2024-09-16T13:46:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=6cc64bd5fbeefcf4c89c709e348395a0 2024-09-16T13:46:18.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary9 2024-09-16T13:46:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=a6b301d12a951ae657d322ecb88b2980 2024-09-16T13:46:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary1 2024-09-16T13:46:18.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=521bce6970c14b30c58b393fa4308b45 2024-09-16T13:46:18.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary4 2024-09-16T13:46:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=d5a9b5c625c8be3a9808954d9abe3471 2024-09-16T13:46:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary3 2024-09-16T13:46:18.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary2 2024-09-16T13:46:18.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:18.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:18.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:46:18.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary2 2024-09-16T13:46:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary7 2024-09-16T13:46:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:46:19.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary7 2024-09-16T13:46:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary5 2024-09-16T13:46:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:46:19.042 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary5 2024-09-16T13:46:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary6 2024-09-16T13:46:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:19.084 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:46:19.085 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary6 2024-09-16T13:46:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary1 2024-09-16T13:46:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:19.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:46:19.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary1 2024-09-16T13:46:19.254 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary9 2024-09-16T13:46:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:46:19.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary9 2024-09-16T13:46:19.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary10 2024-09-16T13:46:19.280 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:19.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:19.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:46:19.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary10 2024-09-16T13:46:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_md5=2ccef09934bba109bacf1c60bef7fa13 2024-09-16T13:46:19.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster1 device unmap -t nbd mirror/image-primary8 2024-09-16T13:46:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary4 2024-09-16T13:46:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:46:19.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary4 2024-09-16T13:46:19.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary3 2024-09-16T13:46:19.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:19.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:19.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:46:19.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary3 2024-09-16T13:46:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ demote_image cluster1 mirror image-primary8 2024-09-16T13:46:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:46:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster1 mirror image demote mirror/image-primary8 2024-09-16T13:46:20.296 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:46:20.286+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c299bd40 2024-09-16T13:46:20.296 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:46:20.290+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8c8c76ea0 2024-09-16T13:46:20.297 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:46:20.290+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8de10aa90 2024-09-16T13:46:20.297 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:46:20.290+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8e1c1edd0 2024-09-16T13:46:20.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:46:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:46:20.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:46:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+unknown 2024-09-16T13:46:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:46:20.346 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:20.347 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:46:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+unknown 2024-09-16T13:46:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+unknown 2024-09-16T13:46:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:46:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:46:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:46:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:46:21.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:46:21.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:46:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:46:21.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:46:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:46:21.329 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:46:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image demoted to non-primary 2024-09-16T13:46:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+unknown 2024-09-16T13:46:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:46:21.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:21.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:46:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+unknown 2024-09-16T13:46:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:46:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:21.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:46:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+unknown 2024-09-16T13:46:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:46:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+unknown 2024-09-16T13:46:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:46:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:46:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:21.353 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:21.354 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:46:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+unknown 2024-09-16T13:46:21.355 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:46:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+unknown 2024-09-16T13:46:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:46:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:46:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:21.356 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:21.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:46:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+unknown 2024-09-16T13:46:21.364 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:46:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:21.365 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:46:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-16T13:46:21.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-16T13:46:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-16T13:46:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:46:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:46:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:46:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T13:46:21.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T13:46:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:21.619 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T13:46:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:46:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 6a9ca88b-8209-4d10-ad07-334e9e0a37da 2024-09-16T13:46:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:46:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:46:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:18 2024-09-16T13:46:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:46:21.727 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:46:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:21' 2024-09-16T13:46:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:46:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 4c7bec10-21b0-4c4a-9fcd-9eafb063806b 2024-09-16T13:46:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:46:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:46:21.729 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:18 2024-09-16T13:46:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:46:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:46:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:21' 2024-09-16T13:46:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:21.730 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:46:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 6a8d0386-ae02-4545-9d78-161debb7a66b 2024-09-16T13:46:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:46:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:46:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:18 2024-09-16T13:46:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:46:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:46:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:21' 2024-09-16T13:46:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:46:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:46:22.032 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:46:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:46:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:46:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:46:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-16T13:46:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:46:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T13:46:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-16T13:46:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:46:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:22.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:22.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T13:46:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-16T13:46:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:46:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:22.357 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-16T13:46:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:46:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T13:46:22.358 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:22.359 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T13:46:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-16T13:46:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:46:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:22.361 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-16T13:46:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:22.362 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T13:46:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:46:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T13:46:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-16T13:46:22.371 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:46:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:22.372 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T13:46:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:46:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 6a68c4a7-d310-4320-ab9d-39bbe0aa88e6 2024-09-16T13:46:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:46:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:46:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:18 2024-09-16T13:46:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:46:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:46:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2142208.0,"last_snapshot_bytes":2592768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494164,"remote_snapshot_timestamp":1726494164,"replay_state":"idle"} 2024-09-16T13:46:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:45:53' 2024-09-16T13:46:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:46:22.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: bf98e067-5114-462a-827a-e4b70ac5d828 2024-09-16T13:46:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:46:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:46:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:18 2024-09-16T13:46:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:46:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image demoted 2024-09-16T13:46:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:22' 2024-09-16T13:46:22.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:46:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:46:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:46:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:46:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: ef925c50-dcd8-411f-9652-6d0180c073f0 2024-09-16T13:46:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:46:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:46:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:18 2024-09-16T13:46:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:46:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:46:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2166784.0,"last_snapshot_bytes":2584576,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494146,"remote_snapshot_timestamp":1726494146,"replay_state":"idle"} 2024-09-16T13:46:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:45:53' 2024-09-16T13:46:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:22.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:46:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:46:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b1662ec3-461a-4192-a479-02fc73ad41c0 2024-09-16T13:46:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:46:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:46:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:18 2024-09-16T13:46:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:46:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:46:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2168832.0,"last_snapshot_bytes":2641920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494160,"remote_snapshot_timestamp":1726494160,"replay_state":"idle"} 2024-09-16T13:46:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:45:53' 2024-09-16T13:46:22.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:46:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:46:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:46:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:46:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:46:22.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 9f5f2075-de41-4691-9994-66fe545d5690 2024-09-16T13:46:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:46:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:46:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:18 2024-09-16T13:46:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:46:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:46:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2142208.0,"last_snapshot_bytes":2629632,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494152,"remote_snapshot_timestamp":1726494152,"replay_state":"idle"} 2024-09-16T13:46:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:45:53' 2024-09-16T13:46:22.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:46:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 9853a33d-aadd-415b-a589-62bc6312de0f 2024-09-16T13:46:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:46:22.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:46:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:18 2024-09-16T13:46:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:46:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:46:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":2148352.0,"last_snapshot_bytes":2588672,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494156,"remote_snapshot_timestamp":1726494156,"replay_state":"idle"} 2024-09-16T13:46:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:45:53' 2024-09-16T13:46:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:46:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 7cc84c68-2260-4571-ae7c-be71801ff8ef 2024-09-16T13:46:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:46:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:46:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:22.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:18 2024-09-16T13:46:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:46:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:46:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":1755136.0,"last_snapshot_bytes":1769472,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494161,"remote_snapshot_timestamp":1726494161,"replay_state":"idle"} 2024-09-16T13:46:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:45:53' 2024-09-16T13:46:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:46:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:46:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:46:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:46:22.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 1 2024-09-16T13:46:22.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:22.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 2 2024-09-16T13:46:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+unknown '' '' 2024-09-16T13:46:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+unknown '' '' 2024-09-16T13:46:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:46:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:46:24.037 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T13:46:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T13:46:24.039 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+unknown '' '' 2024-09-16T13:46:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:46:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:24.040 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T13:46:24.117 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:46:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 6a8d0386-ae02-4545-9d78-161debb7a66b 2024-09-16T13:46:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23 2024-09-16T13:46:24.118 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:46:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23' 2024-09-16T13:46:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:24.120 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:46:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:46:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 6a9ca88b-8209-4d10-ad07-334e9e0a37da 2024-09-16T13:46:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23 2024-09-16T13:46:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:46:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23' 2024-09-16T13:46:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:46:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:46:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 4c7bec10-21b0-4c4a-9fcd-9eafb063806b 2024-09-16T13:46:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23 2024-09-16T13:46:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:46:24.122 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23' 2024-09-16T13:46:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:46:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:46:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:46:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:46:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:46:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:46:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:46:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:46:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:46:24.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:46:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:46:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:46:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:46:24.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:46:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:46:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:46:24.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:46:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:46:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:24.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:46:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:46:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:46:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:46:24.130 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:46:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:46:24.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:46:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+unknown '2024-09-16 13:46:23' 2024-09-16T13:46:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:46:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:46:23' 2024-09-16T13:46:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:24.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:46:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:24.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+unknown '2024-09-16 13:46:23' 2024-09-16T13:46:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:46:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:46:23' 2024-09-16T13:46:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+unknown '2024-09-16 13:46:23' 2024-09-16T13:46:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:46:24.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:46:23' 2024-09-16T13:46:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:46:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:24.153 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:46:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary49f5f2075-de41-4691-9994-66fe545d5690up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23' 2024-09-16T13:46:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:46:24.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T13:46:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary49f5f2075-de41-4691-9994-66fe545d5690up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23' 2024-09-16T13:46:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:46:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary49f5f2075-de41-4691-9994-66fe545d5690up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23' 2024-09-16T13:46:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:46:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T13:46:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T13:46:24.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:24.379 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T13:46:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:24.380 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T13:46:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T13:46:24.384 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:46:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:24.385 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:46:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:46:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:46:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:46:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary7 up+unknown 2024-09-16T13:46:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:46:24.389 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:46:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:46:24.390 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:24.391 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:24.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:24.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:24.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary5 up+unknown 2024-09-16T13:46:24.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:24.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:24.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:46:24.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:24.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:46:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary2 up+unknown 2024-09-16T13:46:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:46:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:24.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:46:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+unknown '' '' 2024-09-16T13:46:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:46:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:24.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+unknown '' '' 2024-09-16T13:46:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:46:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T13:46:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:24.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T13:46:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+unknown '' '' 2024-09-16T13:46:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:46:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T13:46:24.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+unknown '' '' 2024-09-16T13:46:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:46:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:24.448 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T13:46:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+unknown '' '' 2024-09-16T13:46:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:24.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:46:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:24.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+unknown '' '' 2024-09-16T13:46:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:46:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+unknown '' '' 2024-09-16T13:46:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:46:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T13:46:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:24.452 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T13:46:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:24.453 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T13:46:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:46:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: bf98e067-5114-462a-827a-e4b70ac5d828 2024-09-16T13:46:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23 2024-09-16T13:46:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:24.540 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:46:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23' 2024-09-16T13:46:24.541 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:24.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:46:24.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 9f5f2075-de41-4691-9994-66fe545d5690 2024-09-16T13:46:24.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:24.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23 2024-09-16T13:46:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:46:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23' 2024-09-16T13:46:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:24.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:46:24.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:46:24.544 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:46:24.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:46:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:46:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:46:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b1662ec3-461a-4192-a479-02fc73ad41c0 2024-09-16T13:46:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:24.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23 2024-09-16T13:46:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:46:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23' 2024-09-16T13:46:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:46:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 9853a33d-aadd-415b-a589-62bc6312de0f 2024-09-16T13:46:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23 2024-09-16T13:46:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:24.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:46:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23' 2024-09-16T13:46:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:46:24.548 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:46:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: ef925c50-dcd8-411f-9652-6d0180c073f0 2024-09-16T13:46:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23 2024-09-16T13:46:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:46:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23' 2024-09-16T13:46:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:46:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:46:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:46:24.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:46:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:46:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:46:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:46:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:46:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:46:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:46:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:46:24.552 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:46:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:46:24.553 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:46:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:46:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:46:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:46:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:46:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:46:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:46:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:46:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:46:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:46:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:46:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:46:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:46:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:46:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:46:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 6a68c4a7-d310-4320-ab9d-39bbe0aa88e6 2024-09-16T13:46:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23 2024-09-16T13:46:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:46:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23' 2024-09-16T13:46:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:46:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 7cc84c68-2260-4571-ae7c-be71801ff8ef 2024-09-16T13:46:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23 2024-09-16T13:46:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:46:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23' 2024-09-16T13:46:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:46:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:46:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:46:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:46:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+unknown '2024-09-16 13:46:23' 2024-09-16T13:46:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:46:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:46:23' 2024-09-16T13:46:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:46:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:46:24.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:46:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:46:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:46:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:24.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:46:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:46:24.564 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:46:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:46:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:46:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+unknown '2024-09-16 13:46:23' 2024-09-16T13:46:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:24.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:46:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:46:23' 2024-09-16T13:46:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+unknown '2024-09-16 13:46:23' 2024-09-16T13:46:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:46:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:46:24.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:46:23' 2024-09-16T13:46:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:46:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:46:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:46:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:46:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+unknown '2024-09-16 13:46:23' 2024-09-16T13:46:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:46:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:46:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:46:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:46:23' 2024-09-16T13:46:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:46:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:46:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:46:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:46:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:46:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:46:24.571 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:46:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:46:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:46:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:46:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+unknown '2024-09-16 13:46:23' 2024-09-16T13:46:24.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:46:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:46:23' 2024-09-16T13:46:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:46:24.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:46:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:46:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:46:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+unknown '2024-09-16 13:46:23' 2024-09-16T13:46:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:46:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:46:23' 2024-09-16T13:46:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:24.576 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+unknown '2024-09-16 13:46:23' 2024-09-16T13:46:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:46:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:46:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:46:23' 2024-09-16T13:46:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:46:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:24.577 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:46:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary49f5f2075-de41-4691-9994-66fe545d5690up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23' 2024-09-16T13:46:24.854 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:46:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary49f5f2075-de41-4691-9994-66fe545d5690up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23' 2024-09-16T13:46:24.855 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:46:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary49f5f2075-de41-4691-9994-66fe545d5690up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23' 2024-09-16T13:46:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:46:24.856 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T13:46:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T13:46:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary49f5f2075-de41-4691-9994-66fe545d5690up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23' 2024-09-16T13:46:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:46:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T13:46:24.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T13:46:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary49f5f2075-de41-4691-9994-66fe545d5690up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23' 2024-09-16T13:46:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:46:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T13:46:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary49f5f2075-de41-4691-9994-66fe545d5690up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23' 2024-09-16T13:46:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:46:24.860 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T13:46:24.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:24.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:24.861 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T13:46:24.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:24.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T13:46:24.862 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T13:46:24.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:24.863 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T13:46:24.865 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:46:24.866 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:46:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:46:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:46:24.867 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T13:46:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:24.868 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary8 up+unknown 2024-09-16T13:46:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:46:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:24.869 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:46:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary6 up+unknown 2024-09-16T13:46:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:46:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:24.871 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:46:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T13:46:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:46:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:46:24.872 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary3 up+unknown 2024-09-16T13:46:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:46:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:24.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:46:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:46:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:46:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:46:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:46:24.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:46:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:24.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary9 up+unknown 2024-09-16T13:46:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:46:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:46:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:24.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary1 up+unknown 2024-09-16T13:46:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:46:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:46:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:46:24.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:24.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:24.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary10 up+unknown 2024-09-16T13:46:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:46:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:24.881 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:46:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary49f5f2075-de41-4691-9994-66fe545d5690up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+unknownremote image is not primary42544257mirror.0smithi1722024-09-16 13:46:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+unknownremote image is not primary2024-09-16 13:46:23' 2024-09-16T13:46:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:46:24.911 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T13:46:24.917 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T13:46:24.922 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:46:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:46:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster2 mirror image-primary4 up+unknown 2024-09-16T13:46:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:46:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:46:24.926 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:46:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary7 up+unknown '' '' 2024-09-16T13:46:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:46:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary2 up+unknown '' '' 2024-09-16T13:46:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary5 up+unknown '' '' 2024-09-16T13:46:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:46:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:46:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary7 2024-09-16T13:46:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary2 2024-09-16T13:46:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:25.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary5 2024-09-16T13:46:25.458 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:46:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 6a9ca88b-8209-4d10-ad07-334e9e0a37da 2024-09-16T13:46:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23 2024-09-16T13:46:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:46:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23' 2024-09-16T13:46:25.459 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:46:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 6a8d0386-ae02-4545-9d78-161debb7a66b 2024-09-16T13:46:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23 2024-09-16T13:46:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:46:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23' 2024-09-16T13:46:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:25.461 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:46:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:46:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 4c7bec10-21b0-4c4a-9fcd-9eafb063806b 2024-09-16T13:46:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23 2024-09-16T13:46:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:46:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23' 2024-09-16T13:46:25.462 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:46:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:46:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:46:25.463 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:46:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:46:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:46:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:46:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:25.464 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:46:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:46:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:46:25.465 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:25.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:25.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:46:25.466 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:46:25.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:46:25.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:46:25.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:25.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:46:25.467 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:46:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:46:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:46:25.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:46:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:46:25.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:46:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:46:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:46:25.470 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:46:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:46:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:46:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary7 up+unknown '2024-09-16 13:46:23' 2024-09-16T13:46:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:46:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:46:23' 2024-09-16T13:46:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary5 up+unknown '2024-09-16 13:46:23' 2024-09-16T13:46:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:46:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:46:23' 2024-09-16T13:46:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:46:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:46:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary2 up+unknown '2024-09-16 13:46:23' 2024-09-16T13:46:25.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:46:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:46:23' 2024-09-16T13:46:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:25.475 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:46:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary49f5f2075-de41-4691-9994-66fe545d5690up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23' 2024-09-16T13:46:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:46:25.601 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T13:46:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary49f5f2075-de41-4691-9994-66fe545d5690up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23' 2024-09-16T13:46:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:46:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T13:46:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T13:46:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:25.606 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:46:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T13:46:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:25.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:46:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:25.608 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary7 2024-09-16T13:46:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:46:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:46:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary7 2024-09-16T13:46:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:46:25.610 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:46:25.611 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary5 2024-09-16T13:46:25.612 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:46:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:46:25.613 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary5 2024-09-16T13:46:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary49f5f2075-de41-4691-9994-66fe545d5690up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23' 2024-09-16T13:46:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:46:25.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T13:46:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:25.791 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T13:46:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:46:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:46:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:25.796 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary2 2024-09-16T13:46:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:46:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:46:25.797 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary2 2024-09-16T13:46:25.873 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary3 up+unknown '' '' 2024-09-16T13:46:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:46:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary8 up+unknown '' '' 2024-09-16T13:46:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.874 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:46:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary6 up+unknown '' '' 2024-09-16T13:46:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:46:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:25.875 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary3 2024-09-16T13:46:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary8 2024-09-16T13:46:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:25.876 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary6 2024-09-16T13:46:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary9 up+unknown '' '' 2024-09-16T13:46:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary1 up+unknown '' '' 2024-09-16T13:46:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:46:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.877 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:46:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary1 2024-09-16T13:46:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:25.878 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary9 2024-09-16T13:46:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary10 up+unknown '' '' 2024-09-16T13:46:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:46:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:25.879 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:25.880 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary10 2024-09-16T13:46:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster2 mirror image-primary4 up+unknown '' '' 2024-09-16T13:46:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:46:25.927 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:46:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:46:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:25.928 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror image status mirror/image-primary4 2024-09-16T13:46:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:46:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: ef925c50-dcd8-411f-9652-6d0180c073f0 2024-09-16T13:46:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:25.952 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23 2024-09-16T13:46:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:46:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23' 2024-09-16T13:46:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:46:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b1662ec3-461a-4192-a479-02fc73ad41c0 2024-09-16T13:46:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:25.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23 2024-09-16T13:46:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:46:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23' 2024-09-16T13:46:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:46:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:46:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:46:25.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:46:25.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:46:25.959 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:46:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:46:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:46:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:25.960 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:46:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:46:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 7cc84c68-2260-4571-ae7c-be71801ff8ef 2024-09-16T13:46:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:25.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23 2024-09-16T13:46:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:46:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23' 2024-09-16T13:46:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:46:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 9853a33d-aadd-415b-a589-62bc6312de0f 2024-09-16T13:46:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23 2024-09-16T13:46:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:46:25.962 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23' 2024-09-16T13:46:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:46:25.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:46:25.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:25.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:46:25.964 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:46:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:46:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:46:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:46:25.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:46:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:46:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:46:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:46:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:46:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:46:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:46:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:46:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:25.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:46:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:46:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary1 up+unknown '2024-09-16 13:46:23' 2024-09-16T13:46:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:46:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:46:23' 2024-09-16T13:46:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:46:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:46:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:46:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:25.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:46:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:46:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:46:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:46:25.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:46:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: bf98e067-5114-462a-827a-e4b70ac5d828 2024-09-16T13:46:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23 2024-09-16T13:46:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:46:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:25.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23' 2024-09-16T13:46:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:46:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:46:25.973 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:46:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:46:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:46:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:46:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:46:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:46:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary9 up+unknown '2024-09-16 13:46:23' 2024-09-16T13:46:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:46:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:46:23' 2024-09-16T13:46:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:46:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 6a68c4a7-d310-4320-ab9d-39bbe0aa88e6 2024-09-16T13:46:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23 2024-09-16T13:46:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:46:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23' 2024-09-16T13:46:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary6 up+unknown '2024-09-16 13:46:23' 2024-09-16T13:46:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:46:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:46:23' 2024-09-16T13:46:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:46:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:25.978 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:46:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:46:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:46:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:46:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:46:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:46:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary8 up+unknown '2024-09-16 13:46:23' 2024-09-16T13:46:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:46:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:46:23' 2024-09-16T13:46:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:46:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:46:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:46:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:46:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:46:25.983 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:46:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:46:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:46:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:46:25.985 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:46:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary3 up+unknown '2024-09-16 13:46:23' 2024-09-16T13:46:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.987 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:46:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:46:23' 2024-09-16T13:46:25.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:46:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary10 up+unknown '2024-09-16 13:46:23' 2024-09-16T13:46:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:46:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:46:23' 2024-09-16T13:46:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:25.995 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:46:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary49f5f2075-de41-4691-9994-66fe545d5690up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23' 2024-09-16T13:46:26.148 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:46:26.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T13:46:26.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary49f5f2075-de41-4691-9994-66fe545d5690up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23' 2024-09-16T13:46:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:46:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T13:46:26.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary49f5f2075-de41-4691-9994-66fe545d5690up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23' 2024-09-16T13:46:26.199 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:46:26.200 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T13:46:26.205 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T13:46:26.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:26.212 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:46:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T13:46:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T13:46:26.225 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:46:26.228 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:46:26.229 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:46:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary6 2024-09-16T13:46:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:46:26.231 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:46:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary6 2024-09-16T13:46:26.232 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:46:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary1 2024-09-16T13:46:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:46:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:46:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary1 2024-09-16T13:46:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:46:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary9 2024-09-16T13:46:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:26.234 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:46:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:46:26.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary9 2024-09-16T13:46:26.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary49f5f2075-de41-4691-9994-66fe545d5690up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23' 2024-09-16T13:46:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:46:26.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T13:46:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:26.253 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T13:46:26.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary49f5f2075-de41-4691-9994-66fe545d5690up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23' 2024-09-16T13:46:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:46:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:46:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T13:46:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:46:26.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary8 2024-09-16T13:46:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:46:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:46:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary8 2024-09-16T13:46:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary49f5f2075-de41-4691-9994-66fe545d5690up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23' 2024-09-16T13:46:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:46:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T13:46:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T13:46:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:46:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 9f5f2075-de41-4691-9994-66fe545d5690 2024-09-16T13:46:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:46:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23 2024-09-16T13:46:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:46:26.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster1 2024-09-16T13:46:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+unknown 2024-09-16T13:46:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: remote image is not primary 2024-09-16T13:46:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:46:23' 2024-09-16T13:46:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+unknown' 2024-09-16T13:46:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:26.272 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T13:46:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+unknown 2024-09-16T13:46:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:46:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:46:26.274 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:46:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: remote image is not primary 2024-09-16T13:46:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:46:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:26.276 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:46:26.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary3 2024-09-16T13:46:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:46:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:46:26.278 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary3 2024-09-16T13:46:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:46:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:46:26.279 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:46:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:46:26.281 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:46:26.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:46:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:26.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary10 2024-09-16T13:46:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:46:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:46:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary10 2024-09-16T13:46:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster2 mirror image-primary4 up+unknown '2024-09-16 13:46:23' 2024-09-16T13:46:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:46:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+unknown 2024-09-16T13:46:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:46:23' 2024-09-16T13:46:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:46:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:46:26.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster2 mirror pool status mirror --verbose --format xml 2024-09-16T13:46:26.383 INFO:tasks.ceph.cluster2.osd.1.smithi172.stderr:2024-09-16T13:46:26.374+0000 7fcef5fe8700 -1 reset not still connected to 0x55e8db70a270 2024-09-16T13:46:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='WARNINGOKWARNING1042654268mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary49f5f2075-de41-4691-9994-66fe545d5690up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+unknownremote image is not primary42654268mirror.0smithi1722024-09-16 13:46:23cluster1007e7a43-3bf3-48ed-8e55-39258e720141up+unknownremote image is not primary2024-09-16 13:46:23' 2024-09-16T13:46:26.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:46:26.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T13:46:26.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:46:23' 2024-09-16T13:46:26.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T13:46:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+unknown 2024-09-16T13:46:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+unknown 2024-09-16T13:46:26.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+unknown 2024-09-16T13:46:26.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+unknown 2024-09-16T13:46:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:46:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_image cluster2 mirror image-primary4 2024-09-16T13:46:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:46:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:46:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:46:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local force= 2024-09-16T13:46:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 mirror image promote mirror/image-primary4 2024-09-16T13:46:26.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:46:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:46:26.745 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:46:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:46:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:46:26.748 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:46:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:46:26.752 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:46:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:46:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:46:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:46:26.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary9 2024-09-16T13:46:26.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary2 2024-09-16T13:46:26.770 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:46:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary6 2024-09-16T13:46:26.771 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:46:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary10 2024-09-16T13:46:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:46:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:46:26.778 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:46:26.780 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary1 2024-09-16T13:46:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:46:26.782 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary8 2024-09-16T13:46:26.783 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary7 2024-09-16T13:46:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:46:26.786 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary3 2024-09-16T13:46:26.788 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary5 2024-09-16T13:46:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd0 2024-09-16T13:46:26.957 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:46:26.958 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd0 2024-09-16T13:46:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd1 2024-09-16T13:46:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:46:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd1 2024-09-16T13:46:27.012 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd2 2024-09-16T13:46:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:46:27.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd2 2024-09-16T13:46:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd3 2024-09-16T13:46:27.055 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:46:27.056 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd3 2024-09-16T13:46:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd4 2024-09-16T13:46:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:46:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd4 2024-09-16T13:46:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd5 2024-09-16T13:46:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:46:27.159 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd5 2024-09-16T13:46:27.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd6 2024-09-16T13:46:27.172 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:46:27.173 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd6 2024-09-16T13:46:27.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd7 2024-09-16T13:46:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd7 2024-09-16T13:46:27.213 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:46:27.247 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd8 2024-09-16T13:46:27.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:46:27.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd8 2024-09-16T13:46:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:Image promoted to primary 2024-09-16T13:46:27.764 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ nbd == \n\b\d ]] 2024-09-16T13:46:27.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo rbd --cluster cluster2 device map -t nbd -o try-netlink mirror/image-primary4 2024-09-16T13:46:28.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ dev=/dev/nbd9 2024-09-16T13:46:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sudo md5sum /dev/nbd9 2024-09-16T13:46:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ awk '{print $1}' 2024-09-16T13:46:38.831 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:46:38.822+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e36aa000 2024-09-16T13:46:38.831 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:46:38.822+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e36ab1e0 2024-09-16T13:46:38.837 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:46:38.830+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e5d9cf70 2024-09-16T13:46:38.838 INFO:tasks.ceph.cluster2.osd.2.smithi172.stderr:2024-09-16T13:46:38.830+0000 7f315ffe6700 -1 reset not still connected to 0x55c1e5d9d2b0 2024-09-16T13:48:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=91a6db6b9eb83b19755c4222aceef3a4 2024-09-16T13:48:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd0 2024-09-16T13:48:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=ac46edaaae23672000609a93d1cb63d1 2024-09-16T13:48:03.197 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd1 2024-09-16T13:48:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=d5a9b5c625c8be3a9808954d9abe3471 2024-09-16T13:48:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd5 2024-09-16T13:48:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=a419c17ba3b7aa1e4365fd64e5961b41 2024-09-16T13:48:03.857 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd2 2024-09-16T13:48:04.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=7c0ebb690230d30433c3b0023b29a388 2024-09-16T13:48:04.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd3 2024-09-16T13:48:04.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=521bce6970c14b30c58b393fa4308b45 2024-09-16T13:48:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd9 2024-09-16T13:48:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=bd2d5ff43de51882724594e95a408320 2024-09-16T13:48:04.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd7 2024-09-16T13:48:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=a6b301d12a951ae657d322ecb88b2980 2024-09-16T13:48:04.457 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd6 2024-09-16T13:48:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=2ccef09934bba109bacf1c60bef7fa13 2024-09-16T13:48:04.607 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd8 2024-09-16T13:48:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ promote_md5=6cc64bd5fbeefcf4c89c709e348395a0 2024-09-16T13:48:04.693 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sudo rbd --cluster cluster2 device unmap -t nbd /dev/nbd4 2024-09-16T13:48:05.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ a419c17ba3b7aa1e4365fd64e5961b41 != \a\4\1\9\c\1\7\b\a\3\b\7\a\a\1\e\4\3\6\5\f\d\6\4\e\5\9\6\1\b\4\1 ]] 2024-09-16T13:48:05.473 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ ac46edaaae23672000609a93d1cb63d1 != \a\c\4\6\e\d\a\a\a\e\2\3\6\7\2\0\0\0\6\0\9\a\9\3\d\1\c\b\6\3\d\1 ]] 2024-09-16T13:48:05.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 7c0ebb690230d30433c3b0023b29a388 != \7\c\0\e\b\b\6\9\0\2\3\0\d\3\0\4\3\3\c\3\b\0\0\2\3\b\2\9\a\3\8\8 ]] 2024-09-16T13:48:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ a6b301d12a951ae657d322ecb88b2980 != \a\6\b\3\0\1\d\1\2\a\9\5\1\a\e\6\5\7\d\3\2\2\e\c\b\8\8\b\2\9\8\0 ]] 2024-09-16T13:48:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:48:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 139888 2024-09-16T13:48:05.516 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ d5a9b5c625c8be3a9808954d9abe3471 != \d\5\a\9\b\5\c\6\2\5\c\8\b\e\3\a\9\8\0\8\9\5\4\d\9\a\b\e\3\4\7\1 ]] 2024-09-16T13:48:05.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ bd2d5ff43de51882724594e95a408320 != \b\d\2\d\5\f\f\4\3\d\e\5\1\8\8\2\7\2\4\5\9\4\e\9\5\a\4\0\8\3\2\0 ]] 2024-09-16T13:48:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 91a6db6b9eb83b19755c4222aceef3a4 != \9\1\a\6\d\b\6\b\9\e\b\8\3\b\1\9\7\5\5\c\4\2\2\2\a\c\e\e\f\3\a\4 ]] 2024-09-16T13:48:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:48:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 139890 2024-09-16T13:48:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:48:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 139892 2024-09-16T13:48:05.672 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 2ccef09934bba109bacf1c60bef7fa13 != \2\c\c\e\f\0\9\9\3\4\b\b\a\1\0\9\b\a\c\f\1\c\6\0\b\e\f\7\f\a\1\3 ]] 2024-09-16T13:48:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 521bce6970c14b30c58b393fa4308b45 != \5\2\1\b\c\e\6\9\7\0\c\1\4\b\3\0\c\5\8\b\3\9\3\f\a\4\3\0\8\b\4\5 ]] 2024-09-16T13:48:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:48:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 139894 2024-09-16T13:48:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:48:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 139895 2024-09-16T13:48:05.686 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:48:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 139897 2024-09-16T13:48:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:48:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 139899 2024-09-16T13:48:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:48:05.687 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 139900 2024-09-16T13:48:05.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ [[ 6cc64bd5fbeefcf4c89c709e348395a0 != \6\c\c\6\4\b\d\5\f\b\e\e\f\c\f\4\c\8\9\c\7\0\9\e\3\4\8\3\9\5\a\0 ]] 2024-09-16T13:48:05.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for pid in ${COMPARE_PIDS[@]} 2024-09-16T13:48:05.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait 139903 2024-09-16T13:48:05.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ (( RET != 0 )) 2024-09-16T13:48:05.688 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:48:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary1 2024-09-16T13:48:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary1 up+replaying 2024-09-16T13:48:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:48:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:48:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:48:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:48:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:48:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary1 up+replaying '' '' 2024-09-16T13:48:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:48:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:48:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:48:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:48:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:48:06.768 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary1 2024-09-16T13:48:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary1: 2024-09-16T13:48:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: ef925c50-dcd8-411f-9652-6d0180c073f0 2024-09-16T13:48:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:48:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"} 2024-09-16T13:48:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:48:06.852 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:47:53 2024-09-16T13:48:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:48:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:48:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:48:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:48:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:48:03' 2024-09-16T13:48:06.853 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:48:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:48:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:48:06.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"} 2024-09-16T13:48:06.966 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:48:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:48:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:48:06.968 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:48:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:48:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:48:06.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:48:06.972 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:48:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:47:53' 2024-09-16T13:48:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary1 up+replaying '2024-09-16 13:47:53' 2024-09-16T13:48:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:48:07.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:47:53' 2024-09-16T13:48:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:48:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:48:07.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:48:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK1042544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary1ef925c50-dcd8-411f-9652-6d0180c073f0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70193152.0,"last_snapshot_bytes":70193152,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:03image-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70201344.0,"last_snapshot_bytes":70201344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:03image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65609728.0,"last_snapshot_bytes":65609728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:03image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62205952.0,"last_snapshot_bytes":62205952,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:03image-primary49f5f2075-de41-4691-9994-66fe545d5690up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70123520.0,"last_snapshot_bytes":70123520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494387,"remote_snapshot_timestamp":1726494387,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:03image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70164480.0,"last_snapshot_bytes":70164480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:03image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70103040.0,"last_snapshot_bytes":70103040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:03image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":126849024.0,"last_snapshot_bytes":126849024,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:03image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70012928.0,"last_snapshot_bytes":70012928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:03image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72159232.0,"last_snapshot_bytes":72159232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:03' 2024-09-16T13:48:07.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:48:07.151 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/last_update' 2024-09-16T13:48:07.536 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:47:53' 2024-09-16T13:48:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary1'\'']/state' 2024-09-16T13:48:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:48:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:48:07.543 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:48:07.545 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:48:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary1 2024-09-16T13:48:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:48:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:07.546 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:48:07.547 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary1 2024-09-16T13:48:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary1 2024-09-16T13:48:09.243 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:48:09.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary1 2024-09-16T13:48:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary1 2024-09-16T13:48:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:48:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:48:09.250 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary1 2024-09-16T13:48:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:09.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary1 2024-09-16T13:48:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:48:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary2 2024-09-16T13:48:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary2 up+replaying 2024-09-16T13:48:09.298 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:48:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:48:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:48:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:48:09.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:48:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary2 up+replaying '' '' 2024-09-16T13:48:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:10.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:48:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:48:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:48:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:48:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:48:10.301 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary2 2024-09-16T13:48:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary2: 2024-09-16T13:48:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 4c7bec10-21b0-4c4a-9fcd-9eafb063806b 2024-09-16T13:48:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:48:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65609728.0,"last_snapshot_bytes":65609728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"} 2024-09-16T13:48:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:48:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:47:53 2024-09-16T13:48:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:48:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:48:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:48:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:48:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:48:08' 2024-09-16T13:48:10.400 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:48:10.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:48:10.402 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:48:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65609728.0,"last_snapshot_bytes":65609728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"} 2024-09-16T13:48:10.405 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:48:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:48:10.406 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:48:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:48:10.407 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:48:10.409 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:48:10.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:48:10.410 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:48:10.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:47:53' 2024-09-16T13:48:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary2 up+replaying '2024-09-16 13:47:53' 2024-09-16T13:48:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:48:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:47:53' 2024-09-16T13:48:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:48:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:48:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:48:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK942544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70201344.0,"last_snapshot_bytes":70201344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:08image-primary24c7bec10-21b0-4c4a-9fcd-9eafb063806bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":65609728.0,"last_snapshot_bytes":65609728,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:08image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62205952.0,"last_snapshot_bytes":62205952,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:08image-primary49f5f2075-de41-4691-9994-66fe545d5690up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70123520.0,"last_snapshot_bytes":70123520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494387,"remote_snapshot_timestamp":1726494387,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:08image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70164480.0,"last_snapshot_bytes":70164480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:08image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70103040.0,"last_snapshot_bytes":70103040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:08image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":126849024.0,"last_snapshot_bytes":126849024,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:08image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70012928.0,"last_snapshot_bytes":70012928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:08image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72159232.0,"last_snapshot_bytes":72159232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:08' 2024-09-16T13:48:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:48:10.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/last_update' 2024-09-16T13:48:10.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:47:53' 2024-09-16T13:48:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary2'\'']/state' 2024-09-16T13:48:10.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:48:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:48:10.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:48:10.582 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:48:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary2 2024-09-16T13:48:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:48:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:10.583 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:48:10.584 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary2 2024-09-16T13:48:10.650 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary2 2024-09-16T13:48:12.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:48:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary2 2024-09-16T13:48:12.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary2 2024-09-16T13:48:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:48:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary2 2024-09-16T13:48:12.271 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:48:12.318 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary2 2024-09-16T13:48:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:48:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary3 2024-09-16T13:48:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary3 up+replaying 2024-09-16T13:48:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:12.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:48:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:48:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:48:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:48:12.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:48:12.384 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:48:12.372+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T13:48:12.385 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:48:12.376+0000 7ff4f6946700 -1 reset not still connected to 0x557565589ba0 2024-09-16T13:48:12.385 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:48:12.376+0000 7ff4f6946700 -1 reset not still connected to 0x557565589e10 2024-09-16T13:48:13.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary3 up+replaying '' '' 2024-09-16T13:48:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:48:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:48:13.323 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:48:13.324 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary3 2024-09-16T13:48:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary3: 2024-09-16T13:48:13.413 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: bf98e067-5114-462a-827a-e4b70ac5d828 2024-09-16T13:48:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:48:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62205952.0,"last_snapshot_bytes":62205952,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"} 2024-09-16T13:48:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:48:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:47:53 2024-09-16T13:48:13.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:48:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:48:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:48:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:48:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:48:08' 2024-09-16T13:48:13.415 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:48:13.416 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:48:13.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:48:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62205952.0,"last_snapshot_bytes":62205952,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"} 2024-09-16T13:48:13.418 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:48:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:48:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:48:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:48:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:48:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:48:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:48:13.424 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:48:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:47:53' 2024-09-16T13:48:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary3 up+replaying '2024-09-16 13:47:53' 2024-09-16T13:48:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:48:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:47:53' 2024-09-16T13:48:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:48:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:48:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:48:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK842544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70201344.0,"last_snapshot_bytes":70201344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:08image-primary3bf98e067-5114-462a-827a-e4b70ac5d828up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":62205952.0,"last_snapshot_bytes":62205952,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:08image-primary49f5f2075-de41-4691-9994-66fe545d5690up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70123520.0,"last_snapshot_bytes":70123520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494387,"remote_snapshot_timestamp":1726494387,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:08image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70164480.0,"last_snapshot_bytes":70164480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:08image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70103040.0,"last_snapshot_bytes":70103040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:08image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":126849024.0,"last_snapshot_bytes":126849024,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:08image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70012928.0,"last_snapshot_bytes":70012928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:08image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72159232.0,"last_snapshot_bytes":72159232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:08' 2024-09-16T13:48:13.549 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:48:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/last_update' 2024-09-16T13:48:13.556 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:47:53' 2024-09-16T13:48:13.557 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary3'\'']/state' 2024-09-16T13:48:13.562 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:48:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:48:13.563 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:48:13.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:48:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:13.567 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary3 2024-09-16T13:48:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:48:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:48:13.568 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary3 2024-09-16T13:48:13.630 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary3 2024-09-16T13:48:15.277 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:48:15.282 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary3 2024-09-16T13:48:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary3 2024-09-16T13:48:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:48:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:48:15.283 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary3 2024-09-16T13:48:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary3 2024-09-16T13:48:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:48:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary4 2024-09-16T13:48:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary4 up+replaying 2024-09-16T13:48:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:48:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:48:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:48:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:48:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:48:16.196 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:48:16.184+0000 7f61b1855700 -1 reset not still connected to 0x557b3e267110 2024-09-16T13:48:16.196 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:48:16.184+0000 7f61b1855700 -1 reset not still connected to 0x557b41432750 2024-09-16T13:48:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary4 up+replaying '' '' 2024-09-16T13:48:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:48:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:48:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:48:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:48:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:48:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary4 2024-09-16T13:48:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary4: 2024-09-16T13:48:16.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 9f5f2075-de41-4691-9994-66fe545d5690 2024-09-16T13:48:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:48:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70123520.0,"last_snapshot_bytes":70123520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494387,"remote_snapshot_timestamp":1726494387,"replay_state":"idle"} 2024-09-16T13:48:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:48:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:47:53 2024-09-16T13:48:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:48:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:48:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:48:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:48:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:48:13' 2024-09-16T13:48:16.440 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:48:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:48:16.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:48:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70123520.0,"last_snapshot_bytes":70123520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494387,"remote_snapshot_timestamp":1726494387,"replay_state":"idle"} 2024-09-16T13:48:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:48:16.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:48:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:48:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:48:16.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:48:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:48:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:48:16.450 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:48:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:47:53' 2024-09-16T13:48:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary4 up+replaying '2024-09-16 13:47:53' 2024-09-16T13:48:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:48:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:47:53' 2024-09-16T13:48:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:48:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:48:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:48:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK742544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70201344.0,"last_snapshot_bytes":70201344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:13image-primary49f5f2075-de41-4691-9994-66fe545d5690up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70123520.0,"last_snapshot_bytes":70123520,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494387,"remote_snapshot_timestamp":1726494387,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:13image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70164480.0,"last_snapshot_bytes":70164480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:13image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70103040.0,"last_snapshot_bytes":70103040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:13image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":126849024.0,"last_snapshot_bytes":126849024,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:13image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70012928.0,"last_snapshot_bytes":70012928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:13image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72159232.0,"last_snapshot_bytes":72159232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:13' 2024-09-16T13:48:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:48:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/last_update' 2024-09-16T13:48:16.827 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:47:53' 2024-09-16T13:48:16.828 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary4'\'']/state' 2024-09-16T13:48:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:48:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:48:16.835 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:48:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:48:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary4 2024-09-16T13:48:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:48:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:48:16.838 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary4 2024-09-16T13:48:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary4 2024-09-16T13:48:17.038 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:48:17.024+0000 7f61b1855700 -1 reset not still connected to 0x557b1f47fd40 2024-09-16T13:48:17.042 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:48:17.032+0000 7f61b1855700 -1 reset not still connected to 0x557b22af0d00 2024-09-16T13:48:17.042 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:48:17.032+0000 7f61b1855700 -1 reset not still connected to 0x557b22af1ad0 2024-09-16T13:48:17.042 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:48:17.032+0000 7f61b1855700 -1 reset not still connected to 0x557b2cdecea0 2024-09-16T13:48:17.042 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:48:17.032+0000 7f61b1855700 -1 reset not still connected to 0x557b38d76410 2024-09-16T13:48:17.042 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:48:17.032+0000 7f61b1855700 -1 reset not still connected to 0x557b3e267110 2024-09-16T13:48:17.042 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:48:17.032+0000 7f61b1855700 -1 reset not still connected to 0x557b41432750 2024-09-16T13:48:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:48:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary4 2024-09-16T13:48:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary4 2024-09-16T13:48:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:48:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:48:19.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary4 2024-09-16T13:48:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:19.340 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary4 2024-09-16T13:48:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:48:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary5 2024-09-16T13:48:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary5 up+replaying 2024-09-16T13:48:19.341 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:48:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:48:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:48:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:48:19.342 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:48:20.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary5 up+replaying '' '' 2024-09-16T13:48:20.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:20.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:20.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:48:20.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:20.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:48:20.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:48:20.343 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:48:20.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:48:20.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary5 2024-09-16T13:48:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary5: 2024-09-16T13:48:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 6a8d0386-ae02-4545-9d78-161debb7a66b 2024-09-16T13:48:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:48:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70164480.0,"last_snapshot_bytes":70164480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"} 2024-09-16T13:48:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:48:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:47:53 2024-09-16T13:48:20.431 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:48:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:48:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:48:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:48:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:48:18' 2024-09-16T13:48:20.432 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:48:20.433 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:48:20.434 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:48:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70164480.0,"last_snapshot_bytes":70164480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"} 2024-09-16T13:48:20.436 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:48:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:48:20.437 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:48:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:48:20.439 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:48:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:48:20.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:48:20.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:48:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:47:53' 2024-09-16T13:48:20.446 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary5 up+replaying '2024-09-16 13:47:53' 2024-09-16T13:48:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:48:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:47:53' 2024-09-16T13:48:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:48:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:48:20.447 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:48:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK642544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70201344.0,"last_snapshot_bytes":70201344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:18image-primary56a8d0386-ae02-4545-9d78-161debb7a66bup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70164480.0,"last_snapshot_bytes":70164480,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:18image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70103040.0,"last_snapshot_bytes":70103040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:18image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":126849024.0,"last_snapshot_bytes":126849024,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:18image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70012928.0,"last_snapshot_bytes":70012928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:18image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72159232.0,"last_snapshot_bytes":72159232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:47:53cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:18' 2024-09-16T13:48:20.565 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:48:20.566 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/last_update' 2024-09-16T13:48:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:47:53' 2024-09-16T13:48:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary5'\'']/state' 2024-09-16T13:48:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:48:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:48:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:48:20.580 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:48:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary5 2024-09-16T13:48:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:48:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:48:20.581 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary5 2024-09-16T13:48:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary5 2024-09-16T13:48:22.965 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:48:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary5 2024-09-16T13:48:22.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:22.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:22.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary5 2024-09-16T13:48:22.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:48:22.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:48:22.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary5 2024-09-16T13:48:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:23.013 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary5 2024-09-16T13:48:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:48:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary6 2024-09-16T13:48:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary6 up+replaying 2024-09-16T13:48:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:48:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:23.015 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:48:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:48:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:48:23.016 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:48:23.403 INFO:tasks.ceph.cluster1.osd.2.smithi018.stderr:2024-09-16T13:48:23.392+0000 7f61b1855700 -1 reset not still connected to 0x557b41432750 2024-09-16T13:48:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary6 up+replaying '' '' 2024-09-16T13:48:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:48:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:24.017 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:48:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:48:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:48:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:48:24.018 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary6 2024-09-16T13:48:24.108 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary6: 2024-09-16T13:48:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 9853a33d-aadd-415b-a589-62bc6312de0f 2024-09-16T13:48:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:48:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70103040.0,"last_snapshot_bytes":70103040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"} 2024-09-16T13:48:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:48:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:48:23 2024-09-16T13:48:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:48:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:48:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:48:24.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:48:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:48:23' 2024-09-16T13:48:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:48:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:48:24.111 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:48:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70103040.0,"last_snapshot_bytes":70103040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"} 2024-09-16T13:48:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:48:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:48:24.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:48:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:48:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:48:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:48:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:48:24.119 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:48:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:48:23' 2024-09-16T13:48:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary6 up+replaying '2024-09-16 13:48:23' 2024-09-16T13:48:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:48:24.123 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:48:23' 2024-09-16T13:48:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:48:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:48:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:48:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK542544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70201344.0,"last_snapshot_bytes":70201344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:48:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:23image-primary69853a33d-aadd-415b-a589-62bc6312de0fup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70103040.0,"last_snapshot_bytes":70103040,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:48:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":126849024.0,"last_snapshot_bytes":126849024,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:48:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70012928.0,"last_snapshot_bytes":70012928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:48:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72159232.0,"last_snapshot_bytes":72159232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:48:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:23' 2024-09-16T13:48:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:48:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/last_update' 2024-09-16T13:48:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:48:23' 2024-09-16T13:48:24.249 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary6'\'']/state' 2024-09-16T13:48:24.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:48:24.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:48:24.257 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:48:24.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:48:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary6 2024-09-16T13:48:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:48:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:48:24.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary6 2024-09-16T13:48:24.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary6 2024-09-16T13:48:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:48:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary6 2024-09-16T13:48:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary6 2024-09-16T13:48:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:48:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:48:26.002 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary6 2024-09-16T13:48:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary6 2024-09-16T13:48:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:48:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary7 2024-09-16T13:48:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary7 up+replaying 2024-09-16T13:48:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:48:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:26.045 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:48:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:48:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:48:26.046 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:48:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary7 up+replaying '' '' 2024-09-16T13:48:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:48:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:48:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:48:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:48:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:48:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary7 2024-09-16T13:48:27.149 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary7: 2024-09-16T13:48:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 6a9ca88b-8209-4d10-ad07-334e9e0a37da 2024-09-16T13:48:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:48:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":126849024.0,"last_snapshot_bytes":126849024,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"} 2024-09-16T13:48:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:48:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:48:23 2024-09-16T13:48:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:48:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:48:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:48:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:48:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:48:23' 2024-09-16T13:48:27.150 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:48:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:48:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:48:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":126849024.0,"last_snapshot_bytes":126849024,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"} 2024-09-16T13:48:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:48:27.155 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:48:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:48:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:48:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:48:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:48:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:48:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:48:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:48:23' 2024-09-16T13:48:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary7 up+replaying '2024-09-16 13:48:23' 2024-09-16T13:48:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:27.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:48:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:48:23' 2024-09-16T13:48:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:48:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:48:27.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:48:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK442544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70201344.0,"last_snapshot_bytes":70201344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:48:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:23image-primary76a9ca88b-8209-4d10-ad07-334e9e0a37daup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":126849024.0,"last_snapshot_bytes":126849024,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:48:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:23image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70012928.0,"last_snapshot_bytes":70012928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:48:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:23image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72159232.0,"last_snapshot_bytes":72159232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:48:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:23' 2024-09-16T13:48:27.299 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:48:27.300 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/last_update' 2024-09-16T13:48:27.306 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:48:23' 2024-09-16T13:48:27.307 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary7'\'']/state' 2024-09-16T13:48:27.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:48:27.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:48:27.314 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:48:27.315 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary7 2024-09-16T13:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:48:27.317 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary7 2024-09-16T13:48:27.376 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary7 2024-09-16T13:48:27.552 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:48:27.540+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1748f0 2024-09-16T13:48:27.552 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:48:27.540+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174d00 2024-09-16T13:48:27.552 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:48:27.540+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T13:48:27.553 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:48:27.544+0000 7ff4f6946700 -1 reset not still connected to 0x55758877d2b0 2024-09-16T13:48:28.988 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:48:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary7 2024-09-16T13:48:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:28.993 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary7 2024-09-16T13:48:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:48:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:48:28.994 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary7 2024-09-16T13:48:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary7 2024-09-16T13:48:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:48:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary8 2024-09-16T13:48:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary8 up+replaying 2024-09-16T13:48:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:48:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:48:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:48:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:48:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:48:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary8 up+replaying '' '' 2024-09-16T13:48:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:30.035 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:48:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:48:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:48:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:48:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:48:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary8 2024-09-16T13:48:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary8: 2024-09-16T13:48:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: b1662ec3-461a-4192-a479-02fc73ad41c0 2024-09-16T13:48:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:48:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70012928.0,"last_snapshot_bytes":70012928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"} 2024-09-16T13:48:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:48:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:48:23 2024-09-16T13:48:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:48:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:48:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:48:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:48:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:48:28' 2024-09-16T13:48:30.125 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:48:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:48:30.126 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:48:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70012928.0,"last_snapshot_bytes":70012928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"} 2024-09-16T13:48:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:48:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:48:30.129 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:48:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:48:30.131 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:48:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:48:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:48:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:48:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:48:23' 2024-09-16T13:48:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary8 up+replaying '2024-09-16 13:48:23' 2024-09-16T13:48:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:48:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:48:23' 2024-09-16T13:48:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:48:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:48:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:48:30.240 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK342544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70201344.0,"last_snapshot_bytes":70201344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:48:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:28image-primary8b1662ec3-461a-4192-a479-02fc73ad41c0up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70012928.0,"last_snapshot_bytes":70012928,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:48:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:28image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72159232.0,"last_snapshot_bytes":72159232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:48:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:28' 2024-09-16T13:48:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:48:30.241 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/last_update' 2024-09-16T13:48:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:48:23' 2024-09-16T13:48:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary8'\'']/state' 2024-09-16T13:48:30.255 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:48:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:48:30.256 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:48:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:48:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:30.258 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary8 2024-09-16T13:48:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:48:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:48:30.259 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary8 2024-09-16T13:48:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary8 2024-09-16T13:48:31.315 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:48:31.308+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174340 2024-09-16T13:48:31.321 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:48:31.308+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1748f0 2024-09-16T13:48:31.321 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:48:31.308+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174d00 2024-09-16T13:48:31.321 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:48:31.308+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T13:48:32.023 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:48:32.028 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary8 2024-09-16T13:48:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary8 2024-09-16T13:48:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:48:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:48:32.029 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary8 2024-09-16T13:48:32.072 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary8 2024-09-16T13:48:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:48:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary9 2024-09-16T13:48:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary9 up+replaying 2024-09-16T13:48:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:48:32.074 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:48:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:48:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:48:32.075 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:48:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary9 up+replaying '' '' 2024-09-16T13:48:33.076 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:48:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:48:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:48:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:48:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:48:33.077 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary9 2024-09-16T13:48:33.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary9: 2024-09-16T13:48:33.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 7cc84c68-2260-4571-ae7c-be71801ff8ef 2024-09-16T13:48:33.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:48:33.160 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72159232.0,"last_snapshot_bytes":72159232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"} 2024-09-16T13:48:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:48:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:48:23 2024-09-16T13:48:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:48:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:48:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:48:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:48:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:48:28' 2024-09-16T13:48:33.161 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:48:33.162 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:48:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:48:33.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72159232.0,"last_snapshot_bytes":72159232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"} 2024-09-16T13:48:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:48:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:48:33.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:48:33.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:48:33.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:48:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:48:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:48:33.171 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:48:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:48:23' 2024-09-16T13:48:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary9 up+replaying '2024-09-16 13:48:23' 2024-09-16T13:48:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:48:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:48:23' 2024-09-16T13:48:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:48:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:48:33.176 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:48:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK242544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70201344.0,"last_snapshot_bytes":70201344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:48:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:28image-primary97cc84c68-2260-4571-ae7c-be71801ff8efup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":72159232.0,"last_snapshot_bytes":72159232,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:48:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:28' 2024-09-16T13:48:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:48:33.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/last_update' 2024-09-16T13:48:33.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:48:23' 2024-09-16T13:48:33.327 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary9'\'']/state' 2024-09-16T13:48:33.334 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:48:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:48:33.335 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:48:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:48:33.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:33.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:33.337 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary9 2024-09-16T13:48:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:48:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:48:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary9 2024-09-16T13:48:33.414 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary9 2024-09-16T13:48:35.047 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:48:35.053 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary9 2024-09-16T13:48:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary9 2024-09-16T13:48:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:48:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:48:35.054 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary9 2024-09-16T13:48:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:35.099 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary9 2024-09-16T13:48:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for j in {1..10} 2024-09-16T13:48:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ IMG=image-primary10 2024-09-16T13:48:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_status_in_pool_dir cluster1 mirror image-primary10 up+replaying 2024-09-16T13:48:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:48:35.100 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:48:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:48:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:48:35.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:48:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_status_in_pool_dir cluster1 mirror image-primary10 up+replaying '' '' 2024-09-16T13:48:36.101 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:48:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local description_pattern= 2024-09-16T13:48:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local service_pattern= 2024-09-16T13:48:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:48:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:48:36.102 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror image status mirror/image-primary10 2024-09-16T13:48:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='image-primary10: 2024-09-16T13:48:36.182 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: global_id: 6a68c4a7-d310-4320-ab9d-39bbe0aa88e6 2024-09-16T13:48:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+replaying 2024-09-16T13:48:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70201344.0,"last_snapshot_bytes":70201344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"} 2024-09-16T13:48:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: service: mirror.0 on smithi172 2024-09-16T13:48:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:48:23 2024-09-16T13:48:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: peer_sites: 2024-09-16T13:48:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: name: cluster2 2024-09-16T13:48:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: state: up+stopped 2024-09-16T13:48:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: description: local image is primary 2024-09-16T13:48:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: last_update: 2024-09-16 13:48:33' 2024-09-16T13:48:36.183 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ state: .*up+replaying' 2024-09-16T13:48:36.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: state: up+replaying 2024-09-16T13:48:36.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^ description: .*' 2024-09-16T13:48:36.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70201344.0,"last_snapshot_bytes":70201344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"} 2024-09-16T13:48:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n '' ']' 2024-09-16T13:48:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:48:36.187 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep '^up+' 2024-09-16T13:48:36.188 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:48:36.189 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep 'service: *mirror..* on ' 2024-09-16T13:48:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout: service: mirror.0 on smithi172 2024-09-16T13:48:36.191 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update 2024-09-16T13:48:36.192 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-16T13:48:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:48:23' 2024-09-16T13:48:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test_mirror_pool_status_verbose cluster1 mirror image-primary10 up+replaying '2024-09-16 13:48:23' 2024-09-16T13:48:36.195 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:48:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local state_pattern=up+replaying 2024-09-16T13:48:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local 'prev_last_update=2024-09-16 13:48:23' 2024-09-16T13:48:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local status 2024-09-16T13:48:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ CEPH_ARGS= 2024-09-16T13:48:36.196 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-16T13:48:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ status='OKOKOK142544257mirror.0smithi17217.2.7-1643-gd75f2382trueOKimage-primary106a68c4a7-d310-4320-ab9d-39bbe0aa88e6up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":70201344.0,"last_snapshot_bytes":70201344,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726494386,"remote_snapshot_timestamp":1726494386,"replay_state":"idle"}42544257mirror.0smithi1722024-09-16 13:48:23cluster2c96c9d39-ddd8-4a68-9e68-fbf3d4f42450up+stoppedlocal image is primary2024-09-16 13:48:33' 2024-09-16T13:48:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local last_update state 2024-09-16T13:48:36.304 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/last_update' 2024-09-16T13:48:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ last_update='2024-09-16 13:48:23' 2024-09-16T13:48:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''image-primary10'\'']/state' 2024-09-16T13:48:36.319 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ state=up+replaying 2024-09-16T13:48:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo up+replaying 2024-09-16T13:48:36.320 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep up+replaying 2024-09-16T13:48:36.321 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:up+replaying 2024-09-16T13:48:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ remove_image cluster2 mirror image-primary10 2024-09-16T13:48:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:48:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:48:36.322 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 snap purge mirror/image-primary10 2024-09-16T13:48:36.387 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster=cluster2 rm mirror/image-primary10 2024-09-16T13:48:36.861 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:48:36.848+0000 7ff4f6946700 -1 reset not still connected to 0x55753d174d00 2024-09-16T13:48:36.861 INFO:tasks.ceph.cluster1.osd.0.smithi018.stderr:2024-09-16T13:48:36.848+0000 7ff4f6946700 -1 reset not still connected to 0x55753d1755f0 2024-09-16T13:48:38.109 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-16T13:48:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ wait_for_image_removal cluster1 mirror image-primary10 2024-09-16T13:48:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pool=mirror 2024-09-16T13:48:38.114 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local image=image-primary10 2024-09-16T13:48:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-16T13:48:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rbd --cluster cluster1 ls mirror 2024-09-16T13:48:38.115 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ grep -wq image-primary10 2024-09-16T13:48:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ return 0 2024-09-16T13:48:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -rf test-primary10 2024-09-16T13:48:38.167 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo OK 2024-09-16T13:48:38.168 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:OK 2024-09-16T13:48:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ cleanup 0 2024-09-16T13:48:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local error_code=0 2024-09-16T13:48:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ set +e 2024-09-16T13:48:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' 0 -ne 0 ']' 2024-09-16T13:48:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -z '' ']' 2024-09-16T13:48:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster instance 2024-09-16T13:48:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ CEPH_ARGS= 2024-09-16T13:48:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-16T13:48:38.953 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:pool 'mirror' does not exist 2024-09-16T13:48:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ CEPH_ARGS= 2024-09-16T13:48:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-16T13:48:39.919 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:pool 'mirror' does not exist 2024-09-16T13:48:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ CEPH_ARGS= 2024-09-16T13:48:39.931 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-16T13:48:41.127 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:pool 'mirror_parent' does not exist 2024-09-16T13:48:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ CEPH_ARGS= 2024-09-16T13:48:41.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-16T13:48:41.909 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:pool 'mirror_parent' does not exist 2024-09-16T13:48:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-16T13:48:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ stop_mirrors cluster1 2024-09-16T13:48:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1 2024-09-16T13:48:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local sig= 2024-09-16T13:48:41.918 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ seq 0 0 2024-09-16T13:48:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:48:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ stop_mirror cluster1:0 '' 2024-09-16T13:48:41.934 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster1:0 2024-09-16T13:48:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local sig= 2024-09-16T13:48:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test -n '' 2024-09-16T13:48:41.935 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pid 2024-09-16T13:48:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ daemon_pid_file cluster1:0 2024-09-16T13:48:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ local cluster=cluster1:0 2024-09-16T13:48:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ local instance 2024-09-16T13:48:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:48:41.936 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ local val=cluster1:0 2024-09-16T13:48:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ local cluster_var_name=cluster 2024-09-16T13:48:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ local instance_var_name=instance 2024-09-16T13:48:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ cluster=cluster1 2024-09-16T13:48:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ instance=0 2024-09-16T13:48:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-16T13:48:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ eval cluster=cluster1 2024-09-16T13:48:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++++ cluster=cluster1 2024-09-16T13:48:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ eval instance=0 2024-09-16T13:48:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++++ instance=0 2024-09-16T13:48:41.937 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-09-16T13:48:41.970 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-16T13:48:41.971 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ cat /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-16T13:48:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ pid=27135 2024-09-16T13:48:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 27135 ']' 2024-09-16T13:48:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ kill 27135 2024-09-16T13:48:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T13:48:41.984 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:48:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ps auxww 2024-09-16T13:48:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ awk -v pid=27135 '$2 == pid {print; exit 1}' 2024-09-16T13:48:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ break 2024-09-16T13:48:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ps auxww 2024-09-16T13:48:43.136 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ awk -v pid=27135 '$2 == pid {print; exit 1}' 2024-09-16T13:48:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ daemon_asok_file cluster1:0 cluster1 2024-09-16T13:48:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local local_cluster=cluster1:0 2024-09-16T13:48:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local cluster=cluster1 2024-09-16T13:48:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local instance 2024-09-16T13:48:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:48:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local val=cluster1:0 2024-09-16T13:48:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local cluster_var_name=local_cluster 2024-09-16T13:48:43.185 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local instance_var_name=instance 2024-09-16T13:48:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ cluster=cluster1 2024-09-16T13:48:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ instance=0 2024-09-16T13:48:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:48:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ eval local_cluster=cluster1 2024-09-16T13:48:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ local_cluster=cluster1 2024-09-16T13:48:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ eval instance=0 2024-09-16T13:48:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ instance=0 2024-09-16T13:48:43.186 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:48:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:48:43.208 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:48:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ daemon_asok_file cluster1:0 cluster2 2024-09-16T13:48:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local local_cluster=cluster1:0 2024-09-16T13:48:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local cluster=cluster2 2024-09-16T13:48:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local instance 2024-09-16T13:48:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-16T13:48:43.210 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local val=cluster1:0 2024-09-16T13:48:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local cluster_var_name=local_cluster 2024-09-16T13:48:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local instance_var_name=instance 2024-09-16T13:48:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ cluster=cluster1 2024-09-16T13:48:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ instance=0 2024-09-16T13:48:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:48:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ eval local_cluster=cluster1 2024-09-16T13:48:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ local_cluster=cluster1 2024-09-16T13:48:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ eval instance=0 2024-09-16T13:48:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ instance=0 2024-09-16T13:48:43.211 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-16T13:48:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:48:43.235 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-16T13:48:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ daemon_pid_file cluster1:0 2024-09-16T13:48:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local cluster=cluster1:0 2024-09-16T13:48:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local instance 2024-09-16T13:48:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-16T13:48:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local val=cluster1:0 2024-09-16T13:48:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local cluster_var_name=cluster 2024-09-16T13:48:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local instance_var_name=instance 2024-09-16T13:48:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ cluster=cluster1 2024-09-16T13:48:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ instance=0 2024-09-16T13:48:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-16T13:48:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ eval cluster=cluster1 2024-09-16T13:48:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ cluster=cluster1 2024-09-16T13:48:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ eval instance=0 2024-09-16T13:48:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ instance=0 2024-09-16T13:48:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ ceph-conf --cluster cluster1 --name client.mirror.0 'pid file' 2024-09-16T13:48:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-16T13:48:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.pid 2024-09-16T13:48:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-16T13:48:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ stop_mirrors cluster2 2024-09-16T13:48:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2 2024-09-16T13:48:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local sig= 2024-09-16T13:48:43.265 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ seq 0 0 2024-09-16T13:48:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-16T13:48:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ stop_mirror cluster2:0 '' 2024-09-16T13:48:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local cluster=cluster2:0 2024-09-16T13:48:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local sig= 2024-09-16T13:48:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test -n '' 2024-09-16T13:48:43.267 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ local pid 2024-09-16T13:48:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ daemon_pid_file cluster2:0 2024-09-16T13:48:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ local cluster=cluster2:0 2024-09-16T13:48:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ local instance 2024-09-16T13:48:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T13:48:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ local val=cluster2:0 2024-09-16T13:48:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ local cluster_var_name=cluster 2024-09-16T13:48:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ local instance_var_name=instance 2024-09-16T13:48:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ cluster=cluster2 2024-09-16T13:48:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ instance=0 2024-09-16T13:48:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ '[' 0 = cluster2:0 ']' 2024-09-16T13:48:43.269 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ eval cluster=cluster2 2024-09-16T13:48:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++++ cluster=cluster2 2024-09-16T13:48:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ eval instance=0 2024-09-16T13:48:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++++ instance=0 2024-09-16T13:48:43.270 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-09-16T13:48:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-16T13:48:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ cat /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-16T13:48:43.294 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ pid=27164 2024-09-16T13:48:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -n 27164 ']' 2024-09-16T13:48:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ kill 27164 2024-09-16T13:48:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ for s in 1 2 4 8 16 32 2024-09-16T13:48:43.295 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ sleep 1 2024-09-16T13:48:44.296 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ps auxww 2024-09-16T13:48:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ awk -v pid=27164 '$2 == pid {print; exit 1}' 2024-09-16T13:48:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ break 2024-09-16T13:48:44.344 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ ps auxww 2024-09-16T13:48:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ awk -v pid=27164 '$2 == pid {print; exit 1}' 2024-09-16T13:48:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ daemon_asok_file cluster2:0 cluster1 2024-09-16T13:48:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local local_cluster=cluster2:0 2024-09-16T13:48:44.393 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local cluster=cluster1 2024-09-16T13:48:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local instance 2024-09-16T13:48:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T13:48:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local val=cluster2:0 2024-09-16T13:48:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local cluster_var_name=local_cluster 2024-09-16T13:48:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local instance_var_name=instance 2024-09-16T13:48:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ cluster=cluster2 2024-09-16T13:48:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ instance=0 2024-09-16T13:48:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T13:48:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ eval local_cluster=cluster2 2024-09-16T13:48:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ local_cluster=cluster2 2024-09-16T13:48:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ eval instance=0 2024-09-16T13:48:44.394 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ instance=0 2024-09-16T13:48:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T13:48:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T13:48:44.417 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T13:48:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ daemon_asok_file cluster2:0 cluster2 2024-09-16T13:48:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local local_cluster=cluster2:0 2024-09-16T13:48:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local cluster=cluster2 2024-09-16T13:48:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local instance 2024-09-16T13:48:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ set_cluster_instance cluster2:0 local_cluster instance 2024-09-16T13:48:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local val=cluster2:0 2024-09-16T13:48:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local cluster_var_name=local_cluster 2024-09-16T13:48:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local instance_var_name=instance 2024-09-16T13:48:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ cluster=cluster2 2024-09-16T13:48:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ instance=0 2024-09-16T13:48:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T13:48:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ eval local_cluster=cluster2 2024-09-16T13:48:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ local_cluster=cluster2 2024-09-16T13:48:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ eval instance=0 2024-09-16T13:48:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ instance=0 2024-09-16T13:48:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'admin socket' 2024-09-16T13:48:44.441 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T13:48:44.442 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.asok 2024-09-16T13:48:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ daemon_pid_file cluster2:0 2024-09-16T13:48:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local cluster=cluster2:0 2024-09-16T13:48:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local instance 2024-09-16T13:48:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ set_cluster_instance cluster2:0 cluster instance 2024-09-16T13:48:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local val=cluster2:0 2024-09-16T13:48:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local cluster_var_name=cluster 2024-09-16T13:48:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ local instance_var_name=instance 2024-09-16T13:48:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ cluster=cluster2 2024-09-16T13:48:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ instance=0 2024-09-16T13:48:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ '[' 0 = cluster2:0 ']' 2024-09-16T13:48:44.444 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ eval cluster=cluster2 2024-09-16T13:48:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ cluster=cluster2 2024-09-16T13:48:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ eval instance=0 2024-09-16T13:48:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ instance=0 2024-09-16T13:48:44.445 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+++ ceph-conf --cluster cluster2 --name client.mirror.0 'pid file' 2024-09-16T13:48:44.468 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:++ echo /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-16T13:48:44.469 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -f /var/run/ceph/rbd-mirror.cluster2-client.mirror.0.pid 2024-09-16T13:48:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' -z 1 ']' 2024-09-16T13:48:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ test '' = /tmp/tmp.VxztfXQF7C 2024-09-16T13:48:44.471 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ rm -Rf /tmp/tmp.VxztfXQF7C 2024-09-16T13:48:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ '[' 0 -eq 0 ']' 2024-09-16T13:48:44.472 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stdout:OK 2024-09-16T13:48:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ echo OK 2024-09-16T13:48:44.474 INFO:tasks.workunit.cluster1.client.mirror.smithi172.stderr:+ exit 0 2024-09-16T13:48:44.475 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T13:48:44.475 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-09-16T13:48:44.648 INFO:tasks.workunit:Stopping ['rbd/compare_mirror_images.sh'] on cluster1.client.mirror... 2024-09-16T13:48:44.649 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-09-16T13:48:44.866 DEBUG:teuthology.parallel:result is None 2024-09-16T13:48:44.866 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-16T13:48:44.881 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-16T13:48:44.881 DEBUG:teuthology.orchestra.run.smithi172:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-16T13:48:44.888 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-16T13:48:44.888 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T13:48:44.900 INFO:tasks.ceph.ceph_manager.cluster2:Canceling any pending splits or merges... 2024-09-16T13:48:44.900 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T13:48:45.215 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T13:48:45.215 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":1933,"fsid":"ba1e2d9d-316e-497e-977f-9f653ec9cac9","created":"2024-09-16T12:21:13.976545+0000","modified":"2024-09-16T13:48:41.773179+0000","last_up_change":"2024-09-16T12:21:23.671417+0000","last_in_change":"2024-09-16T12:21:14.747781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T12:21:25.621457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T12:21:29.312168+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2dfcac63-8190-45d1-983a-dcb5bf33d348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":21348},{"type":"v1","addr":"172.21.15.172:6801","nonce":21348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":21348},{"type":"v1","addr":"172.21.15.172:6803","nonce":21348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":21348},{"type":"v1","addr":"172.21.15.172:6807","nonce":21348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":21348},{"type":"v1","addr":"172.21.15.172:6805","nonce":21348}]},"public_addr":"172.21.15.172:6801/21348","cluster_addr":"172.21.15.172:6803/21348","heartbeat_back_addr":"172.21.15.172:6807/21348","heartbeat_front_addr":"172.21.15.172:6805/21348","state":["exists","up"]},{"osd":1,"uuid":"7e338fac-a366-462f-90e5-f16f57b3706f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":21350},{"type":"v1","addr":"172.21.15.172:6809","nonce":21350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":21350},{"type":"v1","addr":"172.21.15.172:6811","nonce":21350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":21350},{"type":"v1","addr":"172.21.15.172:6816","nonce":21350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":21350},{"type":"v1","addr":"172.21.15.172:6813","nonce":21350}]},"public_addr":"172.21.15.172:6809/21350","cluster_addr":"172.21.15.172:6811/21350","heartbeat_back_addr":"172.21.15.172:6816/21350","heartbeat_front_addr":"172.21.15.172:6813/21350","state":["exists","up"]},{"osd":2,"uuid":"5987d4f0-55de-4238-adc4-ca6c9b283f18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6815","nonce":21349},{"type":"v1","addr":"172.21.15.172:6817","nonce":21349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":21349},{"type":"v1","addr":"172.21.15.172:6819","nonce":21349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":21349},{"type":"v1","addr":"172.21.15.172:6823","nonce":21349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":21349},{"type":"v1","addr":"172.21.15.172:6821","nonce":21349}]},"public_addr":"172.21.15.172:6817/21349","cluster_addr":"172.21.15.172:6819/21349","heartbeat_back_addr":"172.21.15.172:6823/21349","heartbeat_front_addr":"172.21.15.172:6821/21349","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:21:19.928750+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:21:19.906399+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:21:20.080070+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6568,"length":2},{"begin":6572,"length":1},{"begin":6574,"length":1},{"begin":6586,"length":1},{"begin":6589,"length":3},{"begin":6594,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:48:45.228 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-16T13:48:45.228 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T13:48:45.519 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T13:48:45.523 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-09-16T13:48:45.533 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":4551,"stamp":"2024-09-16T13:48:45.498028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2754904,"kb_used_data":18024,"kb_used_omap":4,"kb_used_meta":2720443,"kb_avail":280360616,"statfs":{"total":289910292480,"available":287089270784,"internally_reserved":0,"allocated":18456576,"data_stored":9569064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":2785733640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-887278404,"num_objects":-271,"num_object_clones":-8,"num_object_copies":-542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-271,"num_whiteouts":0,"num_read":-1060593,"num_read_kb":-57738183,"num_write":-170823,"num_write_kb":-51347773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-24,"num_bytes_recovered":-62849024,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.731207"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1931,"reported_epoch":1931,"state":"active+clean","last_fresh":"2024-09-16T13:48:37.784322+0000","last_change":"2024-09-16T12:21:32.730202+0000","last_active":"2024-09-16T13:48:37.784322+0000","last_peered":"2024-09-16T13:48:37.784322+0000","last_clean":"2024-09-16T13:48:37.784322+0000","last_became_active":"2024-09-16T12:21:30.717557+0000","last_became_peered":"2024-09-16T12:21:30.717557+0000","last_unstale":"2024-09-16T13:48:37.784322+0000","last_undegraded":"2024-09-16T13:48:37.784322+0000","last_fullsized":"2024-09-16T13:48:37.784322+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:30.984888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087664800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1931,"reported_epoch":1931,"state":"active+clean","last_fresh":"2024-09-16T13:48:37.792300+0000","last_change":"2024-09-16T12:21:32.730032+0000","last_active":"2024-09-16T13:48:37.792300+0000","last_peered":"2024-09-16T13:48:37.792300+0000","last_clean":"2024-09-16T13:48:37.792300+0000","last_became_active":"2024-09-16T12:21:30.718053+0000","last_became_peered":"2024-09-16T12:21:30.718053+0000","last_unstale":"2024-09-16T13:48:37.792300+0000","last_undegraded":"2024-09-16T13:48:37.792300+0000","last_fullsized":"2024-09-16T13:48:37.792300+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:30:39.950801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000705296,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1931,"reported_epoch":1931,"state":"active+clean","last_fresh":"2024-09-16T13:48:37.784222+0000","last_change":"2024-09-16T12:21:32.730441+0000","last_active":"2024-09-16T13:48:37.784222+0000","last_peered":"2024-09-16T13:48:37.784222+0000","last_clean":"2024-09-16T13:48:37.784222+0000","last_became_active":"2024-09-16T12:21:30.718170+0000","last_became_peered":"2024-09-16T12:21:30.718170+0000","last_unstale":"2024-09-16T13:48:37.784222+0000","last_undegraded":"2024-09-16T13:48:37.784222+0000","last_fullsized":"2024-09-16T13:48:37.784222+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:29:35.927185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010672889999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1931,"reported_epoch":1931,"state":"active+clean","last_fresh":"2024-09-16T13:48:37.782721+0000","last_change":"2024-09-16T12:21:32.729794+0000","last_active":"2024-09-16T13:48:37.782721+0000","last_peered":"2024-09-16T13:48:37.782721+0000","last_clean":"2024-09-16T13:48:37.782721+0000","last_became_active":"2024-09-16T12:21:30.722079+0000","last_became_peered":"2024-09-16T12:21:30.722079+0000","last_unstale":"2024-09-16T13:48:37.782721+0000","last_undegraded":"2024-09-16T13:48:37.782721+0000","last_fullsized":"2024-09-16T13:48:37.782721+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:01:14.270858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051077700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1933,"reported_epoch":1932,"state":"active+clean","last_fresh":"2024-09-16T13:48:40.102176+0000","last_change":"2024-09-16T12:21:32.727153+0000","last_active":"2024-09-16T13:48:40.102176+0000","last_peered":"2024-09-16T13:48:40.102176+0000","last_clean":"2024-09-16T13:48:40.102176+0000","last_became_active":"2024-09-16T12:21:30.721610+0000","last_became_peered":"2024-09-16T12:21:30.721610+0000","last_unstale":"2024-09-16T13:48:40.102176+0000","last_undegraded":"2024-09-16T13:48:40.102176+0000","last_fullsized":"2024-09-16T13:48:40.102176+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:43:18.475061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014060990000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1932,"reported_epoch":1932,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.783352+0000","last_change":"2024-09-16T12:21:32.719565+0000","last_active":"2024-09-16T13:48:39.783352+0000","last_peered":"2024-09-16T13:48:39.783352+0000","last_clean":"2024-09-16T13:48:39.783352+0000","last_became_active":"2024-09-16T12:21:30.724004+0000","last_became_peered":"2024-09-16T12:21:30.724004+0000","last_unstale":"2024-09-16T13:48:39.783352+0000","last_undegraded":"2024-09-16T13:48:39.783352+0000","last_fullsized":"2024-09-16T13:48:39.783352+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:05.243834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010506230000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1932,"reported_epoch":1932,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.784963+0000","last_change":"2024-09-16T12:21:32.719558+0000","last_active":"2024-09-16T13:48:39.784963+0000","last_peered":"2024-09-16T13:48:39.784963+0000","last_clean":"2024-09-16T13:48:39.784963+0000","last_became_active":"2024-09-16T12:21:30.721621+0000","last_became_peered":"2024-09-16T12:21:30.721621+0000","last_unstale":"2024-09-16T13:48:39.784963+0000","last_undegraded":"2024-09-16T13:48:39.784963+0000","last_fullsized":"2024-09-16T13:48:39.784963+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:53:01.393711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001027781,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1932,"reported_epoch":1931,"state":"active+clean","last_fresh":"2024-09-16T13:48:37.784444+0000","last_change":"2024-09-16T12:21:32.730634+0000","last_active":"2024-09-16T13:48:37.784444+0000","last_peered":"2024-09-16T13:48:37.784444+0000","last_clean":"2024-09-16T13:48:37.784444+0000","last_became_active":"2024-09-16T12:21:30.717785+0000","last_became_peered":"2024-09-16T12:21:30.717785+0000","last_unstale":"2024-09-16T13:48:37.784444+0000","last_undegraded":"2024-09-16T13:48:37.784444+0000","last_fullsized":"2024-09-16T13:48:37.784444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:30:09.189422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012262880000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":1995,"reported_epoch":1931,"state":"active+clean","last_fresh":"2024-09-16T13:48:37.784602+0000","last_change":"2024-09-16T12:21:26.698782+0000","last_active":"2024-09-16T13:48:37.784602+0000","last_peered":"2024-09-16T13:48:37.784602+0000","last_clean":"2024-09-16T13:48:37.784602+0000","last_became_active":"2024-09-16T12:21:26.696232+0000","last_became_peered":"2024-09-16T12:21:26.696232+0000","last_unstale":"2024-09-16T13:48:37.784602+0000","last_undegraded":"2024-09-16T13:48:37.784602+0000","last_fullsized":"2024-09-16T13:48:37.784602+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:25.674706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:25.674706+0000","last_clean_scrub_stamp":"2024-09-16T12:21:25.674706+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:28.926290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739422,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914688,"kb_used_data":5620,"kb_used_omap":1,"kb_used_meta":909054,"kb_avail":93457152,"statfs":{"total":96636764160,"available":95700123648,"internally_reserved":0,"allocated":5754880,"data_stored":2796261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":930871640},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:48:24 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":0.66700000000000004,"15min":0.63600000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.184},"max":{"1min":14.237,"5min":14.237,"15min":14.237},"last":0.56699999999999995},{"interface":"front","average":{"1min":2.0299999999999998,"5min":0.90700000000000003,"15min":0.73999999999999999},"min":{"1min":0.30199999999999999,"5min":0.25,"15min":0.23100000000000001},"max":{"1min":13.818,"5min":13.818,"15min":13.818},"last":0.51300000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.61899999999999999,"15min":0.68799999999999994},"min":{"1min":0.26300000000000001,"5min":0.26200000000000001,"15min":0.215},"max":{"1min":2.7570000000000001,"5min":2.7570000000000001,"15min":10.692},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.68000000000000005,"15min":0.749},"min":{"1min":0.27600000000000002,"5min":0.27400000000000002,"15min":0.22900000000000001},"max":{"1min":2.7090000000000001,"5min":2.8380000000000001,"15min":10.747},"last":0.73399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739424,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919560,"kb_used_data":6192,"kb_used_omap":1,"kb_used_meta":896958,"kb_avail":93452280,"statfs":{"total":96636764160,"available":95695134720,"internally_reserved":0,"allocated":6340608,"data_stored":3382448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918485336},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:48:26 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.70099999999999996,"15min":0.68899999999999995},"min":{"1min":0.22,"5min":0.22,"15min":0.182},"max":{"1min":5.641,"5min":5.641,"15min":7.1600000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.65800000000000003,"15min":0.61699999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":3.5110000000000001,"5min":3.552,"15min":3.585},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Sep 16 13:48:26 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.63,"15min":0.63100000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":3.4740000000000002,"5min":3.4740000000000002,"15min":4.6950000000000003},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.64800000000000002,"15min":0.67500000000000004},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":3.0350000000000001,"5min":3.0350000000000001,"15min":3.5369999999999999},"last":0.72999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739422,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":920656,"kb_used_data":6212,"kb_used_omap":1,"kb_used_meta":914430,"kb_avail":93451184,"statfs":{"total":96636764160,"available":95694012416,"internally_reserved":0,"allocated":6361088,"data_stored":3390355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936376664},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":1.101,"15min":0.86499999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.19500000000000001},"max":{"1min":0.94799999999999995,"5min":57.963000000000001,"15min":57.963000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.66300000000000003,"15min":0.63},"min":{"1min":0.22600000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.1820000000000004,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":0.45100000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:48:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":1.026,"15min":0.92000000000000004},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":2.0699999999999998,"5min":17.372,"15min":17.372},"last":0.624},{"interface":"front","average":{"1min":0.623,"5min":0.70899999999999996,"15min":0.68700000000000006},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.4139999999999999,"5min":7.3049999999999997,"15min":7.3049999999999997},"last":0.56399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:48:45.534 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T13:48:45.836 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T13:48:45.839 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-09-16T13:48:45.849 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":4551,"stamp":"2024-09-16T13:48:45.498028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2754904,"kb_used_data":18024,"kb_used_omap":4,"kb_used_meta":2720443,"kb_avail":280360616,"statfs":{"total":289910292480,"available":287089270784,"internally_reserved":0,"allocated":18456576,"data_stored":9569064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":2785733640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-887278404,"num_objects":-271,"num_object_clones":-8,"num_object_copies":-542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-271,"num_whiteouts":0,"num_read":-1060593,"num_read_kb":-57738183,"num_write":-170823,"num_write_kb":-51347773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-24,"num_bytes_recovered":-62849024,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.731207"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1931,"reported_epoch":1931,"state":"active+clean","last_fresh":"2024-09-16T13:48:37.784322+0000","last_change":"2024-09-16T12:21:32.730202+0000","last_active":"2024-09-16T13:48:37.784322+0000","last_peered":"2024-09-16T13:48:37.784322+0000","last_clean":"2024-09-16T13:48:37.784322+0000","last_became_active":"2024-09-16T12:21:30.717557+0000","last_became_peered":"2024-09-16T12:21:30.717557+0000","last_unstale":"2024-09-16T13:48:37.784322+0000","last_undegraded":"2024-09-16T13:48:37.784322+0000","last_fullsized":"2024-09-16T13:48:37.784322+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:30.984888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087664800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1931,"reported_epoch":1931,"state":"active+clean","last_fresh":"2024-09-16T13:48:37.792300+0000","last_change":"2024-09-16T12:21:32.730032+0000","last_active":"2024-09-16T13:48:37.792300+0000","last_peered":"2024-09-16T13:48:37.792300+0000","last_clean":"2024-09-16T13:48:37.792300+0000","last_became_active":"2024-09-16T12:21:30.718053+0000","last_became_peered":"2024-09-16T12:21:30.718053+0000","last_unstale":"2024-09-16T13:48:37.792300+0000","last_undegraded":"2024-09-16T13:48:37.792300+0000","last_fullsized":"2024-09-16T13:48:37.792300+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:30:39.950801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000705296,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1931,"reported_epoch":1931,"state":"active+clean","last_fresh":"2024-09-16T13:48:37.784222+0000","last_change":"2024-09-16T12:21:32.730441+0000","last_active":"2024-09-16T13:48:37.784222+0000","last_peered":"2024-09-16T13:48:37.784222+0000","last_clean":"2024-09-16T13:48:37.784222+0000","last_became_active":"2024-09-16T12:21:30.718170+0000","last_became_peered":"2024-09-16T12:21:30.718170+0000","last_unstale":"2024-09-16T13:48:37.784222+0000","last_undegraded":"2024-09-16T13:48:37.784222+0000","last_fullsized":"2024-09-16T13:48:37.784222+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:29:35.927185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010672889999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1931,"reported_epoch":1931,"state":"active+clean","last_fresh":"2024-09-16T13:48:37.782721+0000","last_change":"2024-09-16T12:21:32.729794+0000","last_active":"2024-09-16T13:48:37.782721+0000","last_peered":"2024-09-16T13:48:37.782721+0000","last_clean":"2024-09-16T13:48:37.782721+0000","last_became_active":"2024-09-16T12:21:30.722079+0000","last_became_peered":"2024-09-16T12:21:30.722079+0000","last_unstale":"2024-09-16T13:48:37.782721+0000","last_undegraded":"2024-09-16T13:48:37.782721+0000","last_fullsized":"2024-09-16T13:48:37.782721+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:01:14.270858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051077700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1933,"reported_epoch":1932,"state":"active+clean","last_fresh":"2024-09-16T13:48:40.102176+0000","last_change":"2024-09-16T12:21:32.727153+0000","last_active":"2024-09-16T13:48:40.102176+0000","last_peered":"2024-09-16T13:48:40.102176+0000","last_clean":"2024-09-16T13:48:40.102176+0000","last_became_active":"2024-09-16T12:21:30.721610+0000","last_became_peered":"2024-09-16T12:21:30.721610+0000","last_unstale":"2024-09-16T13:48:40.102176+0000","last_undegraded":"2024-09-16T13:48:40.102176+0000","last_fullsized":"2024-09-16T13:48:40.102176+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:43:18.475061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014060990000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1932,"reported_epoch":1932,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.783352+0000","last_change":"2024-09-16T12:21:32.719565+0000","last_active":"2024-09-16T13:48:39.783352+0000","last_peered":"2024-09-16T13:48:39.783352+0000","last_clean":"2024-09-16T13:48:39.783352+0000","last_became_active":"2024-09-16T12:21:30.724004+0000","last_became_peered":"2024-09-16T12:21:30.724004+0000","last_unstale":"2024-09-16T13:48:39.783352+0000","last_undegraded":"2024-09-16T13:48:39.783352+0000","last_fullsized":"2024-09-16T13:48:39.783352+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:05.243834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010506230000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1932,"reported_epoch":1932,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.784963+0000","last_change":"2024-09-16T12:21:32.719558+0000","last_active":"2024-09-16T13:48:39.784963+0000","last_peered":"2024-09-16T13:48:39.784963+0000","last_clean":"2024-09-16T13:48:39.784963+0000","last_became_active":"2024-09-16T12:21:30.721621+0000","last_became_peered":"2024-09-16T12:21:30.721621+0000","last_unstale":"2024-09-16T13:48:39.784963+0000","last_undegraded":"2024-09-16T13:48:39.784963+0000","last_fullsized":"2024-09-16T13:48:39.784963+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:53:01.393711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001027781,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1932,"reported_epoch":1931,"state":"active+clean","last_fresh":"2024-09-16T13:48:37.784444+0000","last_change":"2024-09-16T12:21:32.730634+0000","last_active":"2024-09-16T13:48:37.784444+0000","last_peered":"2024-09-16T13:48:37.784444+0000","last_clean":"2024-09-16T13:48:37.784444+0000","last_became_active":"2024-09-16T12:21:30.717785+0000","last_became_peered":"2024-09-16T12:21:30.717785+0000","last_unstale":"2024-09-16T13:48:37.784444+0000","last_undegraded":"2024-09-16T13:48:37.784444+0000","last_fullsized":"2024-09-16T13:48:37.784444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:30:09.189422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012262880000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":1995,"reported_epoch":1931,"state":"active+clean","last_fresh":"2024-09-16T13:48:37.784602+0000","last_change":"2024-09-16T12:21:26.698782+0000","last_active":"2024-09-16T13:48:37.784602+0000","last_peered":"2024-09-16T13:48:37.784602+0000","last_clean":"2024-09-16T13:48:37.784602+0000","last_became_active":"2024-09-16T12:21:26.696232+0000","last_became_peered":"2024-09-16T12:21:26.696232+0000","last_unstale":"2024-09-16T13:48:37.784602+0000","last_undegraded":"2024-09-16T13:48:37.784602+0000","last_fullsized":"2024-09-16T13:48:37.784602+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:25.674706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:25.674706+0000","last_clean_scrub_stamp":"2024-09-16T12:21:25.674706+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:28.926290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739422,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914688,"kb_used_data":5620,"kb_used_omap":1,"kb_used_meta":909054,"kb_avail":93457152,"statfs":{"total":96636764160,"available":95700123648,"internally_reserved":0,"allocated":5754880,"data_stored":2796261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":930871640},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:48:24 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":0.66700000000000004,"15min":0.63600000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.184},"max":{"1min":14.237,"5min":14.237,"15min":14.237},"last":0.56699999999999995},{"interface":"front","average":{"1min":2.0299999999999998,"5min":0.90700000000000003,"15min":0.73999999999999999},"min":{"1min":0.30199999999999999,"5min":0.25,"15min":0.23100000000000001},"max":{"1min":13.818,"5min":13.818,"15min":13.818},"last":0.51300000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.61899999999999999,"15min":0.68799999999999994},"min":{"1min":0.26300000000000001,"5min":0.26200000000000001,"15min":0.215},"max":{"1min":2.7570000000000001,"5min":2.7570000000000001,"15min":10.692},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.68000000000000005,"15min":0.749},"min":{"1min":0.27600000000000002,"5min":0.27400000000000002,"15min":0.22900000000000001},"max":{"1min":2.7090000000000001,"5min":2.8380000000000001,"15min":10.747},"last":0.73399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739424,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919560,"kb_used_data":6192,"kb_used_omap":1,"kb_used_meta":896958,"kb_avail":93452280,"statfs":{"total":96636764160,"available":95695134720,"internally_reserved":0,"allocated":6340608,"data_stored":3382448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918485336},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:48:26 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.70099999999999996,"15min":0.68899999999999995},"min":{"1min":0.22,"5min":0.22,"15min":0.182},"max":{"1min":5.641,"5min":5.641,"15min":7.1600000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.65800000000000003,"15min":0.61699999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":3.5110000000000001,"5min":3.552,"15min":3.585},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Sep 16 13:48:26 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.63,"15min":0.63100000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":3.4740000000000002,"5min":3.4740000000000002,"15min":4.6950000000000003},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.64800000000000002,"15min":0.67500000000000004},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":3.0350000000000001,"5min":3.0350000000000001,"15min":3.5369999999999999},"last":0.72999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739422,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":920656,"kb_used_data":6212,"kb_used_omap":1,"kb_used_meta":914430,"kb_avail":93451184,"statfs":{"total":96636764160,"available":95694012416,"internally_reserved":0,"allocated":6361088,"data_stored":3390355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936376664},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":1.101,"15min":0.86499999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.19500000000000001},"max":{"1min":0.94799999999999995,"5min":57.963000000000001,"15min":57.963000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.66300000000000003,"15min":0.63},"min":{"1min":0.22600000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.1820000000000004,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":0.45100000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:48:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":1.026,"15min":0.92000000000000004},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":2.0699999999999998,"5min":17.372,"15min":17.372},"last":0.624},{"interface":"front","average":{"1min":0.623,"5min":0.70899999999999996,"15min":0.68700000000000006},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.4139999999999999,"5min":7.3049999999999997,"15min":7.3049999999999997},"last":0.56399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:48:45.850 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-16T13:48:45.850 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T13:48:46.142 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T13:48:46.145 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-09-16T13:48:46.156 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":4551,"stamp":"2024-09-16T13:48:45.498028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2754904,"kb_used_data":18024,"kb_used_omap":4,"kb_used_meta":2720443,"kb_avail":280360616,"statfs":{"total":289910292480,"available":287089270784,"internally_reserved":0,"allocated":18456576,"data_stored":9569064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":2785733640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-887278404,"num_objects":-271,"num_object_clones":-8,"num_object_copies":-542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-271,"num_whiteouts":0,"num_read":-1060593,"num_read_kb":-57738183,"num_write":-170823,"num_write_kb":-51347773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-24,"num_bytes_recovered":-62849024,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.731207"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1931,"reported_epoch":1931,"state":"active+clean","last_fresh":"2024-09-16T13:48:37.784322+0000","last_change":"2024-09-16T12:21:32.730202+0000","last_active":"2024-09-16T13:48:37.784322+0000","last_peered":"2024-09-16T13:48:37.784322+0000","last_clean":"2024-09-16T13:48:37.784322+0000","last_became_active":"2024-09-16T12:21:30.717557+0000","last_became_peered":"2024-09-16T12:21:30.717557+0000","last_unstale":"2024-09-16T13:48:37.784322+0000","last_undegraded":"2024-09-16T13:48:37.784322+0000","last_fullsized":"2024-09-16T13:48:37.784322+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:30.984888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087664800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1931,"reported_epoch":1931,"state":"active+clean","last_fresh":"2024-09-16T13:48:37.792300+0000","last_change":"2024-09-16T12:21:32.730032+0000","last_active":"2024-09-16T13:48:37.792300+0000","last_peered":"2024-09-16T13:48:37.792300+0000","last_clean":"2024-09-16T13:48:37.792300+0000","last_became_active":"2024-09-16T12:21:30.718053+0000","last_became_peered":"2024-09-16T12:21:30.718053+0000","last_unstale":"2024-09-16T13:48:37.792300+0000","last_undegraded":"2024-09-16T13:48:37.792300+0000","last_fullsized":"2024-09-16T13:48:37.792300+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:30:39.950801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000705296,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1931,"reported_epoch":1931,"state":"active+clean","last_fresh":"2024-09-16T13:48:37.784222+0000","last_change":"2024-09-16T12:21:32.730441+0000","last_active":"2024-09-16T13:48:37.784222+0000","last_peered":"2024-09-16T13:48:37.784222+0000","last_clean":"2024-09-16T13:48:37.784222+0000","last_became_active":"2024-09-16T12:21:30.718170+0000","last_became_peered":"2024-09-16T12:21:30.718170+0000","last_unstale":"2024-09-16T13:48:37.784222+0000","last_undegraded":"2024-09-16T13:48:37.784222+0000","last_fullsized":"2024-09-16T13:48:37.784222+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:29:35.927185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010672889999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1931,"reported_epoch":1931,"state":"active+clean","last_fresh":"2024-09-16T13:48:37.782721+0000","last_change":"2024-09-16T12:21:32.729794+0000","last_active":"2024-09-16T13:48:37.782721+0000","last_peered":"2024-09-16T13:48:37.782721+0000","last_clean":"2024-09-16T13:48:37.782721+0000","last_became_active":"2024-09-16T12:21:30.722079+0000","last_became_peered":"2024-09-16T12:21:30.722079+0000","last_unstale":"2024-09-16T13:48:37.782721+0000","last_undegraded":"2024-09-16T13:48:37.782721+0000","last_fullsized":"2024-09-16T13:48:37.782721+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:01:14.270858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051077700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1933,"reported_epoch":1932,"state":"active+clean","last_fresh":"2024-09-16T13:48:40.102176+0000","last_change":"2024-09-16T12:21:32.727153+0000","last_active":"2024-09-16T13:48:40.102176+0000","last_peered":"2024-09-16T13:48:40.102176+0000","last_clean":"2024-09-16T13:48:40.102176+0000","last_became_active":"2024-09-16T12:21:30.721610+0000","last_became_peered":"2024-09-16T12:21:30.721610+0000","last_unstale":"2024-09-16T13:48:40.102176+0000","last_undegraded":"2024-09-16T13:48:40.102176+0000","last_fullsized":"2024-09-16T13:48:40.102176+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:43:18.475061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014060990000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1932,"reported_epoch":1932,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.783352+0000","last_change":"2024-09-16T12:21:32.719565+0000","last_active":"2024-09-16T13:48:39.783352+0000","last_peered":"2024-09-16T13:48:39.783352+0000","last_clean":"2024-09-16T13:48:39.783352+0000","last_became_active":"2024-09-16T12:21:30.724004+0000","last_became_peered":"2024-09-16T12:21:30.724004+0000","last_unstale":"2024-09-16T13:48:39.783352+0000","last_undegraded":"2024-09-16T13:48:39.783352+0000","last_fullsized":"2024-09-16T13:48:39.783352+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:05.243834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010506230000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1932,"reported_epoch":1932,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.784963+0000","last_change":"2024-09-16T12:21:32.719558+0000","last_active":"2024-09-16T13:48:39.784963+0000","last_peered":"2024-09-16T13:48:39.784963+0000","last_clean":"2024-09-16T13:48:39.784963+0000","last_became_active":"2024-09-16T12:21:30.721621+0000","last_became_peered":"2024-09-16T12:21:30.721621+0000","last_unstale":"2024-09-16T13:48:39.784963+0000","last_undegraded":"2024-09-16T13:48:39.784963+0000","last_fullsized":"2024-09-16T13:48:39.784963+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:53:01.393711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001027781,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1932,"reported_epoch":1931,"state":"active+clean","last_fresh":"2024-09-16T13:48:37.784444+0000","last_change":"2024-09-16T12:21:32.730634+0000","last_active":"2024-09-16T13:48:37.784444+0000","last_peered":"2024-09-16T13:48:37.784444+0000","last_clean":"2024-09-16T13:48:37.784444+0000","last_became_active":"2024-09-16T12:21:30.717785+0000","last_became_peered":"2024-09-16T12:21:30.717785+0000","last_unstale":"2024-09-16T13:48:37.784444+0000","last_undegraded":"2024-09-16T13:48:37.784444+0000","last_fullsized":"2024-09-16T13:48:37.784444+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:30:09.189422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012262880000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":1995,"reported_epoch":1931,"state":"active+clean","last_fresh":"2024-09-16T13:48:37.784602+0000","last_change":"2024-09-16T12:21:26.698782+0000","last_active":"2024-09-16T13:48:37.784602+0000","last_peered":"2024-09-16T13:48:37.784602+0000","last_clean":"2024-09-16T13:48:37.784602+0000","last_became_active":"2024-09-16T12:21:26.696232+0000","last_became_peered":"2024-09-16T12:21:26.696232+0000","last_unstale":"2024-09-16T13:48:37.784602+0000","last_undegraded":"2024-09-16T13:48:37.784602+0000","last_fullsized":"2024-09-16T13:48:37.784602+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:25.674706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:25.674706+0000","last_clean_scrub_stamp":"2024-09-16T12:21:25.674706+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:28.926290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739422,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914688,"kb_used_data":5620,"kb_used_omap":1,"kb_used_meta":909054,"kb_avail":93457152,"statfs":{"total":96636764160,"available":95700123648,"internally_reserved":0,"allocated":5754880,"data_stored":2796261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":930871640},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:48:24 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":0.66700000000000004,"15min":0.63600000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.184},"max":{"1min":14.237,"5min":14.237,"15min":14.237},"last":0.56699999999999995},{"interface":"front","average":{"1min":2.0299999999999998,"5min":0.90700000000000003,"15min":0.73999999999999999},"min":{"1min":0.30199999999999999,"5min":0.25,"15min":0.23100000000000001},"max":{"1min":13.818,"5min":13.818,"15min":13.818},"last":0.51300000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.61899999999999999,"15min":0.68799999999999994},"min":{"1min":0.26300000000000001,"5min":0.26200000000000001,"15min":0.215},"max":{"1min":2.7570000000000001,"5min":2.7570000000000001,"15min":10.692},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.68000000000000005,"15min":0.749},"min":{"1min":0.27600000000000002,"5min":0.27400000000000002,"15min":0.22900000000000001},"max":{"1min":2.7090000000000001,"5min":2.8380000000000001,"15min":10.747},"last":0.73399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739424,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":919560,"kb_used_data":6192,"kb_used_omap":1,"kb_used_meta":896958,"kb_avail":93452280,"statfs":{"total":96636764160,"available":95695134720,"internally_reserved":0,"allocated":6340608,"data_stored":3382448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918485336},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:48:26 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.70099999999999996,"15min":0.68899999999999995},"min":{"1min":0.22,"5min":0.22,"15min":0.182},"max":{"1min":5.641,"5min":5.641,"15min":7.1600000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.65800000000000003,"15min":0.61699999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":3.5110000000000001,"5min":3.552,"15min":3.585},"last":0.58599999999999997}]},{"osd":2,"last update":"Mon Sep 16 13:48:26 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.63,"15min":0.63100000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":3.4740000000000002,"5min":3.4740000000000002,"15min":4.6950000000000003},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.64800000000000002,"15min":0.67500000000000004},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":3.0350000000000001,"5min":3.0350000000000001,"15min":3.5369999999999999},"last":0.72999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359739422,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":920656,"kb_used_data":6212,"kb_used_omap":1,"kb_used_meta":914430,"kb_avail":93451184,"statfs":{"total":96636764160,"available":95694012416,"internally_reserved":0,"allocated":6361088,"data_stored":3390355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936376664},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":1.101,"15min":0.86499999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.19500000000000001},"max":{"1min":0.94799999999999995,"5min":57.963000000000001,"15min":57.963000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.66300000000000003,"15min":0.63},"min":{"1min":0.22600000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.1820000000000004,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":0.45100000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:48:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":1.026,"15min":0.92000000000000004},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":2.0699999999999998,"5min":17.372,"15min":17.372},"last":0.624},{"interface":"front","average":{"1min":0.623,"5min":0.70899999999999996,"15min":0.68700000000000006},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.4139999999999999,"5min":7.3049999999999997,"15min":7.3049999999999997},"last":0.56399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:48:46.157 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-16T13:48:46.449 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T13:48:46.449 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":1933,"fsid":"ba1e2d9d-316e-497e-977f-9f653ec9cac9","created":"2024-09-16T12:21:13.976545+0000","modified":"2024-09-16T13:48:41.773179+0000","last_up_change":"2024-09-16T12:21:23.671417+0000","last_in_change":"2024-09-16T12:21:14.747781+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T12:21:25.621457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T12:21:29.312168+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2dfcac63-8190-45d1-983a-dcb5bf33d348","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":21348},{"type":"v1","addr":"172.21.15.172:6801","nonce":21348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":21348},{"type":"v1","addr":"172.21.15.172:6803","nonce":21348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":21348},{"type":"v1","addr":"172.21.15.172:6807","nonce":21348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":21348},{"type":"v1","addr":"172.21.15.172:6805","nonce":21348}]},"public_addr":"172.21.15.172:6801/21348","cluster_addr":"172.21.15.172:6803/21348","heartbeat_back_addr":"172.21.15.172:6807/21348","heartbeat_front_addr":"172.21.15.172:6805/21348","state":["exists","up"]},{"osd":1,"uuid":"7e338fac-a366-462f-90e5-f16f57b3706f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":21350},{"type":"v1","addr":"172.21.15.172:6809","nonce":21350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":21350},{"type":"v1","addr":"172.21.15.172:6811","nonce":21350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":21350},{"type":"v1","addr":"172.21.15.172:6816","nonce":21350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":21350},{"type":"v1","addr":"172.21.15.172:6813","nonce":21350}]},"public_addr":"172.21.15.172:6809/21350","cluster_addr":"172.21.15.172:6811/21350","heartbeat_back_addr":"172.21.15.172:6816/21350","heartbeat_front_addr":"172.21.15.172:6813/21350","state":["exists","up"]},{"osd":2,"uuid":"5987d4f0-55de-4238-adc4-ca6c9b283f18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6815","nonce":21349},{"type":"v1","addr":"172.21.15.172:6817","nonce":21349}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":21349},{"type":"v1","addr":"172.21.15.172:6819","nonce":21349}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":21349},{"type":"v1","addr":"172.21.15.172:6823","nonce":21349}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":21349},{"type":"v1","addr":"172.21.15.172:6821","nonce":21349}]},"public_addr":"172.21.15.172:6817/21349","cluster_addr":"172.21.15.172:6819/21349","heartbeat_back_addr":"172.21.15.172:6823/21349","heartbeat_front_addr":"172.21.15.172:6821/21349","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:21:19.928750+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:21:19.906399+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:21:20.080070+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6568,"length":2},{"begin":6572,"length":1},{"begin":6574,"length":1},{"begin":6586,"length":1},{"begin":6589,"length":3},{"begin":6594,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:48:47.464 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-09-16T13:48:47.464 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:48:47.616 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-16T13:48:47.616 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:48:47.616 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-16T13:48:47.629 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-09-16T13:48:47.934 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T13:48:47.950 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-09-16T13:48:47.950 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:48:48.092 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-16T13:48:48.092 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:48:48.092 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-16T13:48:48.105 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-09-16T13:48:48.395 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T13:48:48.410 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-09-16T13:48:48.410 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:48:48.546 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-09-16T13:48:48.546 INFO:teuthology.orchestra.run.smithi172.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:48:48.546 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-09-16T13:48:48.559 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-09-16T13:48:48.855 INFO:teuthology.orchestra.run.smithi172.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T13:48:48.865 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T13:48:49.157 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T13:48:49.160 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-09-16T13:48:49.171 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":4552,"stamp":"2024-09-16T13:48:47.498529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":106,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2738528,"kb_used_data":18044,"kb_used_omap":4,"kb_used_meta":2720443,"kb_avail":280376992,"statfs":{"total":289910292480,"available":287106039808,"internally_reserved":0,"allocated":18477056,"data_stored":9576971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":2785733640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-887278404,"num_objects":-271,"num_object_clones":-8,"num_object_copies":-542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-271,"num_whiteouts":0,"num_read":-1060593,"num_read_kb":-57738183,"num_write":-170823,"num_write_kb":-51347773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-24,"num_bytes_recovered":-62849024,"num_keys_recovered":0,"num_objects_omap":-15,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002559"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1932,"reported_epoch":1932,"state":"active+clean","last_fresh":"2024-09-16T13:48:40.219992+0000","last_change":"2024-09-16T12:21:32.730202+0000","last_active":"2024-09-16T13:48:40.219992+0000","last_peered":"2024-09-16T13:48:40.219992+0000","last_clean":"2024-09-16T13:48:40.219992+0000","last_became_active":"2024-09-16T12:21:30.717557+0000","last_became_peered":"2024-09-16T12:21:30.717557+0000","last_unstale":"2024-09-16T13:48:40.219992+0000","last_undegraded":"2024-09-16T13:48:40.219992+0000","last_fullsized":"2024-09-16T13:48:40.219992+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:33:30.984888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087664800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1932,"reported_epoch":1932,"state":"active+clean","last_fresh":"2024-09-16T13:48:40.226065+0000","last_change":"2024-09-16T12:21:32.730032+0000","last_active":"2024-09-16T13:48:40.226065+0000","last_peered":"2024-09-16T13:48:40.226065+0000","last_clean":"2024-09-16T13:48:40.226065+0000","last_became_active":"2024-09-16T12:21:30.718053+0000","last_became_peered":"2024-09-16T12:21:30.718053+0000","last_unstale":"2024-09-16T13:48:40.226065+0000","last_undegraded":"2024-09-16T13:48:40.226065+0000","last_fullsized":"2024-09-16T13:48:40.226065+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:30:39.950801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000705296,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1932,"reported_epoch":1932,"state":"active+clean","last_fresh":"2024-09-16T13:48:40.227931+0000","last_change":"2024-09-16T12:21:32.730441+0000","last_active":"2024-09-16T13:48:40.227931+0000","last_peered":"2024-09-16T13:48:40.227931+0000","last_clean":"2024-09-16T13:48:40.227931+0000","last_became_active":"2024-09-16T12:21:30.718170+0000","last_became_peered":"2024-09-16T12:21:30.718170+0000","last_unstale":"2024-09-16T13:48:40.227931+0000","last_undegraded":"2024-09-16T13:48:40.227931+0000","last_fullsized":"2024-09-16T13:48:40.227931+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:29:35.927185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010672889999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1932,"reported_epoch":1932,"state":"active+clean","last_fresh":"2024-09-16T13:48:40.217384+0000","last_change":"2024-09-16T12:21:32.729794+0000","last_active":"2024-09-16T13:48:40.217384+0000","last_peered":"2024-09-16T13:48:40.217384+0000","last_clean":"2024-09-16T13:48:40.217384+0000","last_became_active":"2024-09-16T12:21:30.722079+0000","last_became_peered":"2024-09-16T12:21:30.722079+0000","last_unstale":"2024-09-16T13:48:40.217384+0000","last_undegraded":"2024-09-16T13:48:40.217384+0000","last_fullsized":"2024-09-16T13:48:40.217384+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:01:14.270858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051077700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1933,"reported_epoch":1932,"state":"active+clean","last_fresh":"2024-09-16T13:48:40.102176+0000","last_change":"2024-09-16T12:21:32.727153+0000","last_active":"2024-09-16T13:48:40.102176+0000","last_peered":"2024-09-16T13:48:40.102176+0000","last_clean":"2024-09-16T13:48:40.102176+0000","last_became_active":"2024-09-16T12:21:30.721610+0000","last_became_peered":"2024-09-16T12:21:30.721610+0000","last_unstale":"2024-09-16T13:48:40.102176+0000","last_undegraded":"2024-09-16T13:48:40.102176+0000","last_fullsized":"2024-09-16T13:48:40.102176+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:43:18.475061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014060990000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1932,"reported_epoch":1932,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.783352+0000","last_change":"2024-09-16T12:21:32.719565+0000","last_active":"2024-09-16T13:48:39.783352+0000","last_peered":"2024-09-16T13:48:39.783352+0000","last_clean":"2024-09-16T13:48:39.783352+0000","last_became_active":"2024-09-16T12:21:30.724004+0000","last_became_peered":"2024-09-16T12:21:30.724004+0000","last_unstale":"2024-09-16T13:48:39.783352+0000","last_undegraded":"2024-09-16T13:48:39.783352+0000","last_fullsized":"2024-09-16T13:48:39.783352+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:05.243834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010506230000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1932,"reported_epoch":1932,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.784963+0000","last_change":"2024-09-16T12:21:32.719558+0000","last_active":"2024-09-16T13:48:39.784963+0000","last_peered":"2024-09-16T13:48:39.784963+0000","last_clean":"2024-09-16T13:48:39.784963+0000","last_became_active":"2024-09-16T12:21:30.721621+0000","last_became_peered":"2024-09-16T12:21:30.721621+0000","last_unstale":"2024-09-16T13:48:39.784963+0000","last_undegraded":"2024-09-16T13:48:39.784963+0000","last_fullsized":"2024-09-16T13:48:39.784963+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:53:01.393711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001027781,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1933,"reported_epoch":1932,"state":"active+clean","last_fresh":"2024-09-16T13:48:40.214857+0000","last_change":"2024-09-16T12:21:32.730634+0000","last_active":"2024-09-16T13:48:40.214857+0000","last_peered":"2024-09-16T13:48:40.214857+0000","last_clean":"2024-09-16T13:48:40.214857+0000","last_became_active":"2024-09-16T12:21:30.717785+0000","last_became_peered":"2024-09-16T12:21:30.717785+0000","last_unstale":"2024-09-16T13:48:40.214857+0000","last_undegraded":"2024-09-16T13:48:40.214857+0000","last_fullsized":"2024-09-16T13:48:40.214857+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:30:09.189422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012262880000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'52","reported_seq":1996,"reported_epoch":1932,"state":"active+clean","last_fresh":"2024-09-16T13:48:40.205626+0000","last_change":"2024-09-16T12:21:26.698782+0000","last_active":"2024-09-16T13:48:40.205626+0000","last_peered":"2024-09-16T13:48:40.205626+0000","last_clean":"2024-09-16T13:48:40.205626+0000","last_became_active":"2024-09-16T12:21:26.696232+0000","last_became_peered":"2024-09-16T12:21:26.696232+0000","last_unstale":"2024-09-16T13:48:40.205626+0000","last_undegraded":"2024-09-16T13:48:40.205626+0000","last_fullsized":"2024-09-16T13:48:40.205626+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:25.674706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:25.674706+0000","last_clean_scrub_stamp":"2024-09-16T12:21:25.674706+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T16:24:28.926290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739422,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914688,"kb_used_data":5620,"kb_used_omap":1,"kb_used_meta":909054,"kb_avail":93457152,"statfs":{"total":96636764160,"available":95700123648,"internally_reserved":0,"allocated":5754880,"data_stored":2796261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":930871640},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:48:24 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":0.66700000000000004,"15min":0.63600000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.184},"max":{"1min":14.237,"5min":14.237,"15min":14.237},"last":0.56699999999999995},{"interface":"front","average":{"1min":2.0299999999999998,"5min":0.90700000000000003,"15min":0.73999999999999999},"min":{"1min":0.30199999999999999,"5min":0.25,"15min":0.23100000000000001},"max":{"1min":13.818,"5min":13.818,"15min":13.818},"last":0.51300000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.61899999999999999,"15min":0.68799999999999994},"min":{"1min":0.26300000000000001,"5min":0.26200000000000001,"15min":0.215},"max":{"1min":2.7570000000000001,"5min":2.7570000000000001,"15min":10.692},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.68000000000000005,"15min":0.749},"min":{"1min":0.27600000000000002,"5min":0.27400000000000002,"15min":0.22900000000000001},"max":{"1min":2.7090000000000001,"5min":2.8380000000000001,"15min":10.747},"last":0.73399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739425,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":903184,"kb_used_data":6212,"kb_used_omap":1,"kb_used_meta":896958,"kb_avail":93468656,"statfs":{"total":96636764160,"available":95711903744,"internally_reserved":0,"allocated":6361088,"data_stored":3390355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918485336},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:48:26 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.70099999999999996,"15min":0.68899999999999995},"min":{"1min":0.22,"5min":0.22,"15min":0.182},"max":{"1min":5.641,"5min":5.641,"15min":7.1600000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.65800000000000003,"15min":0.61699999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":3.5110000000000001,"5min":3.552,"15min":3.585},"last":0.55100000000000005}]},{"osd":2,"last update":"Mon Sep 16 13:48:26 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.63,"15min":0.63100000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":3.4740000000000002,"5min":3.4740000000000002,"15min":4.6950000000000003},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.64800000000000002,"15min":0.67500000000000004},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":3.0350000000000001,"5min":3.0350000000000001,"15min":3.5369999999999999},"last":0.83799999999999997}]}]},{"osd":0,"up_from":8,"seq":34359739422,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":920656,"kb_used_data":6212,"kb_used_omap":1,"kb_used_meta":914430,"kb_avail":93451184,"statfs":{"total":96636764160,"available":95694012416,"internally_reserved":0,"allocated":6361088,"data_stored":3390355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936376664},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":1.101,"15min":0.86499999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.19500000000000001},"max":{"1min":0.94799999999999995,"5min":57.963000000000001,"15min":57.963000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.66300000000000003,"15min":0.63},"min":{"1min":0.22600000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.1820000000000004,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":0.45100000000000001}]},{"osd":2,"last update":"Mon Sep 16 13:48:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":1.026,"15min":0.92000000000000004},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":2.0699999999999998,"5min":17.372,"15min":17.372},"last":0.624},{"interface":"front","average":{"1min":0.623,"5min":0.70899999999999996,"15min":0.68700000000000006},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.4139999999999999,"5min":7.3049999999999997,"15min":7.3049999999999997},"last":0.56399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:48:49.175 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T12:21:29.693546+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=21, tm_sec=29, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=48, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:48:49.175 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T12:21:29.693546+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=21, tm_sec=29, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=48, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:48:49.175 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T12:21:29.693546+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=21, tm_sec=29, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=48, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:48:49.176 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T12:21:29.693546+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=21, tm_sec=29, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=48, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:48:49.176 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T12:21:29.693546+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=21, tm_sec=29, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=48, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:48:49.176 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T12:21:29.693546+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=21, tm_sec=29, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=48, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:48:49.176 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T12:21:29.693546+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=21, tm_sec=29, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=48, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:48:49.176 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T12:21:29.693546+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=21, tm_sec=29, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=48, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:48:49.176 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-16T12:21:25.674706+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=21, tm_sec=25, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=48, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:48:49.176 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T13:49:09.177 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T13:49:09.478 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T13:49:09.481 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-09-16T13:49:09.491 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":4562,"stamp":"2024-09-16T13:49:07.503599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2738524,"kb_used_data":18052,"kb_used_omap":4,"kb_used_meta":2720443,"kb_avail":280376996,"statfs":{"total":289910292480,"available":287106043904,"internally_reserved":0,"allocated":18485248,"data_stored":9580678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":2785733640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003098"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1943,"reported_epoch":1933,"state":"active+clean","last_fresh":"2024-09-16T13:49:02.966363+0000","last_change":"2024-09-16T13:49:02.966300+0000","last_active":"2024-09-16T13:49:02.966363+0000","last_peered":"2024-09-16T13:49:02.966363+0000","last_clean":"2024-09-16T13:49:02.966363+0000","last_became_active":"2024-09-16T12:21:30.717557+0000","last_became_peered":"2024-09-16T12:21:30.717557+0000","last_unstale":"2024-09-16T13:49:02.966363+0000","last_undegraded":"2024-09-16T13:49:02.966363+0000","last_fullsized":"2024-09-16T13:49:02.966363+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:49:02.966079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:49:02.966079+0000","last_clean_scrub_stamp":"2024-09-16T13:49:02.966079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:42:20.670584+0000","scrub_duration":0.0032263169999999998,"objects_trimmed":0,"snaptrim_duration":0.00087664800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1933,"reported_epoch":1933,"state":"active+clean","last_fresh":"2024-09-16T13:48:42.958830+0000","last_change":"2024-09-16T12:21:32.730032+0000","last_active":"2024-09-16T13:48:42.958830+0000","last_peered":"2024-09-16T13:48:42.958830+0000","last_clean":"2024-09-16T13:48:42.958830+0000","last_became_active":"2024-09-16T12:21:30.718053+0000","last_became_peered":"2024-09-16T12:21:30.718053+0000","last_unstale":"2024-09-16T13:48:42.958830+0000","last_undegraded":"2024-09-16T13:48:42.958830+0000","last_fullsized":"2024-09-16T13:48:42.958830+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000705296,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1933,"reported_epoch":1933,"state":"active+clean","last_fresh":"2024-09-16T13:48:42.958776+0000","last_change":"2024-09-16T12:21:32.730441+0000","last_active":"2024-09-16T13:48:42.958776+0000","last_peered":"2024-09-16T13:48:42.958776+0000","last_clean":"2024-09-16T13:48:42.958776+0000","last_became_active":"2024-09-16T12:21:30.718170+0000","last_became_peered":"2024-09-16T12:21:30.718170+0000","last_unstale":"2024-09-16T13:48:42.958776+0000","last_undegraded":"2024-09-16T13:48:42.958776+0000","last_fullsized":"2024-09-16T13:48:42.958776+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010672889999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1933,"reported_epoch":1933,"state":"active+clean","last_fresh":"2024-09-16T13:48:42.958725+0000","last_change":"2024-09-16T12:21:32.729794+0000","last_active":"2024-09-16T13:48:42.958725+0000","last_peered":"2024-09-16T13:48:42.958725+0000","last_clean":"2024-09-16T13:48:42.958725+0000","last_became_active":"2024-09-16T12:21:30.722079+0000","last_became_peered":"2024-09-16T12:21:30.722079+0000","last_unstale":"2024-09-16T13:48:42.958725+0000","last_undegraded":"2024-09-16T13:48:42.958725+0000","last_fullsized":"2024-09-16T13:48:42.958725+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:21:29.693546+0000","last_clean_scrub_stamp":"2024-09-16T12:21:29.693546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051077700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"1933'3","reported_seq":1945,"reported_epoch":1933,"state":"active+clean","last_fresh":"2024-09-16T13:48:52.554554+0000","last_change":"2024-09-16T13:48:52.554495+0000","last_active":"2024-09-16T13:48:52.554554+0000","last_peered":"2024-09-16T13:48:52.554554+0000","last_clean":"2024-09-16T13:48:52.554554+0000","last_became_active":"2024-09-16T12:21:30.721610+0000","last_became_peered":"2024-09-16T12:21:30.721610+0000","last_unstale":"2024-09-16T13:48:52.554554+0000","last_undegraded":"2024-09-16T13:48:52.554554+0000","last_fullsized":"2024-09-16T13:48:52.554554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"1933'3","last_scrub_stamp":"2024-09-16T13:48:52.554288+0000","last_deep_scrub":"1933'3","last_deep_scrub_stamp":"2024-09-16T13:48:52.554288+0000","last_clean_scrub_stamp":"2024-09-16T13:48:52.554288+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:42:19.324696+0000","scrub_duration":0.0064325509999999999,"objects_trimmed":0,"snaptrim_duration":0.0014060990000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1943,"reported_epoch":1933,"state":"active+clean","last_fresh":"2024-09-16T13:48:53.204630+0000","last_change":"2024-09-16T13:48:53.204500+0000","last_active":"2024-09-16T13:48:53.204630+0000","last_peered":"2024-09-16T13:48:53.204630+0000","last_clean":"2024-09-16T13:48:53.204630+0000","last_became_active":"2024-09-16T12:21:30.724004+0000","last_became_peered":"2024-09-16T12:21:30.724004+0000","last_unstale":"2024-09-16T13:48:53.204630+0000","last_undegraded":"2024-09-16T13:48:53.204630+0000","last_fullsized":"2024-09-16T13:48:53.204630+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:48:53.204127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:48:53.204127+0000","last_clean_scrub_stamp":"2024-09-16T13:48:53.204127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:07.224110+0000","scrub_duration":0.0039067379999999999,"objects_trimmed":0,"snaptrim_duration":0.0010506230000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1943,"reported_epoch":1933,"state":"active+clean","last_fresh":"2024-09-16T13:48:49.186785+0000","last_change":"2024-09-16T13:48:49.186746+0000","last_active":"2024-09-16T13:48:49.186785+0000","last_peered":"2024-09-16T13:48:49.186785+0000","last_clean":"2024-09-16T13:48:49.186785+0000","last_became_active":"2024-09-16T12:21:30.721621+0000","last_became_peered":"2024-09-16T12:21:30.721621+0000","last_unstale":"2024-09-16T13:48:49.186785+0000","last_undegraded":"2024-09-16T13:48:49.186785+0000","last_fullsized":"2024-09-16T13:48:49.186785+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:48:49.186576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:48:49.186576+0000","last_clean_scrub_stamp":"2024-09-16T13:48:49.186576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:08:50.755168+0000","scrub_duration":0.002865397,"objects_trimmed":0,"snaptrim_duration":0.001027781,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"1933'2","reported_seq":1945,"reported_epoch":1933,"state":"active+clean","last_fresh":"2024-09-16T13:48:59.917536+0000","last_change":"2024-09-16T13:48:59.917442+0000","last_active":"2024-09-16T13:48:59.917536+0000","last_peered":"2024-09-16T13:48:59.917536+0000","last_clean":"2024-09-16T13:48:59.917536+0000","last_became_active":"2024-09-16T12:21:30.717785+0000","last_became_peered":"2024-09-16T12:21:30.717785+0000","last_unstale":"2024-09-16T13:48:59.917536+0000","last_undegraded":"2024-09-16T13:48:59.917536+0000","last_fullsized":"2024-09-16T13:48:59.917536+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"1933'2","last_scrub_stamp":"2024-09-16T13:48:59.917109+0000","last_deep_scrub":"1933'2","last_deep_scrub_stamp":"2024-09-16T13:48:59.917109+0000","last_clean_scrub_stamp":"2024-09-16T13:48:59.917109+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:26:12.885514+0000","scrub_duration":0.0069628520000000003,"objects_trimmed":0,"snaptrim_duration":0.0012262880000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"1933'54","reported_seq":2009,"reported_epoch":1933,"state":"active+clean","last_fresh":"2024-09-16T13:48:50.966566+0000","last_change":"2024-09-16T13:48:50.966487+0000","last_active":"2024-09-16T13:48:50.966566+0000","last_peered":"2024-09-16T13:48:50.966566+0000","last_clean":"2024-09-16T13:48:50.966566+0000","last_became_active":"2024-09-16T12:21:26.696232+0000","last_became_peered":"2024-09-16T12:21:26.696232+0000","last_unstale":"2024-09-16T13:48:50.966566+0000","last_undegraded":"2024-09-16T13:48:50.966566+0000","last_fullsized":"2024-09-16T13:48:50.966566+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"1933'54","last_scrub_stamp":"2024-09-16T13:48:50.966345+0000","last_deep_scrub":"1933'54","last_deep_scrub_stamp":"2024-09-16T13:48:50.966345+0000","last_clean_scrub_stamp":"2024-09-16T13:48:50.966345+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:27:41.193137+0000","scrub_duration":0.0090738499999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739426,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914692,"kb_used_data":5628,"kb_used_omap":1,"kb_used_meta":909054,"kb_avail":93457148,"statfs":{"total":96636764160,"available":95700119552,"internally_reserved":0,"allocated":5763072,"data_stored":2799968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":930871640},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:48:24 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":0.66700000000000004,"15min":0.63600000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.184},"max":{"1min":14.237,"5min":14.237,"15min":14.237},"last":0.44600000000000001},{"interface":"front","average":{"1min":2.0299999999999998,"5min":0.90700000000000003,"15min":0.73999999999999999},"min":{"1min":0.30199999999999999,"5min":0.25,"15min":0.23100000000000001},"max":{"1min":13.818,"5min":13.818,"15min":13.818},"last":0.55800000000000005}]},{"osd":1,"last update":"Mon Sep 16 13:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.61899999999999999,"15min":0.68799999999999994},"min":{"1min":0.26300000000000001,"5min":0.26200000000000001,"15min":0.215},"max":{"1min":2.7570000000000001,"5min":2.7570000000000001,"15min":10.692},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.68000000000000005,"15min":0.749},"min":{"1min":0.27600000000000002,"5min":0.27400000000000002,"15min":0.22900000000000001},"max":{"1min":2.7090000000000001,"5min":2.8380000000000001,"15min":10.747},"last":0.59599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359739429,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":903180,"kb_used_data":6212,"kb_used_omap":1,"kb_used_meta":896958,"kb_avail":93468660,"statfs":{"total":96636764160,"available":95711907840,"internally_reserved":0,"allocated":6361088,"data_stored":3390355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918485336},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:48:26 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.70099999999999996,"15min":0.68899999999999995},"min":{"1min":0.22,"5min":0.22,"15min":0.182},"max":{"1min":5.641,"5min":5.641,"15min":7.1600000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.65800000000000003,"15min":0.61699999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":3.5110000000000001,"5min":3.552,"15min":3.585},"last":0.55400000000000005}]},{"osd":2,"last update":"Mon Sep 16 13:48:26 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.63,"15min":0.63100000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":3.4740000000000002,"5min":3.4740000000000002,"15min":4.6950000000000003},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.64800000000000002,"15min":0.67500000000000004},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":3.0350000000000001,"5min":3.0350000000000001,"15min":3.5369999999999999},"last":0.67500000000000004}]}]},{"osd":0,"up_from":8,"seq":34359739426,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":920652,"kb_used_data":6212,"kb_used_omap":1,"kb_used_meta":914430,"kb_avail":93451188,"statfs":{"total":96636764160,"available":95694016512,"internally_reserved":0,"allocated":6361088,"data_stored":3390355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936376664},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:48:20 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":1.101,"15min":0.86499999999999999},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.19500000000000001},"max":{"1min":0.94799999999999995,"5min":57.963000000000001,"15min":57.963000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.66300000000000003,"15min":0.63},"min":{"1min":0.22600000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":5.1820000000000004,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":0.64200000000000002}]},{"osd":2,"last update":"Mon Sep 16 13:48:09 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":1.026,"15min":0.92000000000000004},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":2.0699999999999998,"5min":17.372,"15min":17.372},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.623,"5min":0.70899999999999996,"15min":0.68700000000000006},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.254},"max":{"1min":1.4139999999999999,"5min":7.3049999999999997,"15min":7.3049999999999997},"last":0.74299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:49:09.493 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T12:21:29.693546+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=21, tm_sec=29, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=48, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:49:09.493 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T12:21:29.693546+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=21, tm_sec=29, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=48, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:49:09.493 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T12:21:29.693546+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=21, tm_sec=29, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=48, tm_sec=46, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:49:09.493 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T13:49:29.494 DEBUG:teuthology.orchestra.run.smithi172:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-16T13:49:29.793 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-09-16T13:49:29.796 INFO:teuthology.orchestra.run.smithi172.stderr:dumped all 2024-09-16T13:49:29.807 INFO:teuthology.orchestra.run.smithi172.stdout:{"pg_ready":true,"pg_map":{"version":4573,"stamp":"2024-09-16T13:49:29.509240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2738524,"kb_used_data":18052,"kb_used_omap":4,"kb_used_meta":2720443,"kb_avail":280376996,"statfs":{"total":289910292480,"available":287106043904,"internally_reserved":0,"allocated":18485248,"data_stored":9580678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":2785733640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003087"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1943,"reported_epoch":1933,"state":"active+clean","last_fresh":"2024-09-16T13:49:02.966363+0000","last_change":"2024-09-16T13:49:02.966300+0000","last_active":"2024-09-16T13:49:02.966363+0000","last_peered":"2024-09-16T13:49:02.966363+0000","last_clean":"2024-09-16T13:49:02.966363+0000","last_became_active":"2024-09-16T12:21:30.717557+0000","last_became_peered":"2024-09-16T12:21:30.717557+0000","last_unstale":"2024-09-16T13:49:02.966363+0000","last_undegraded":"2024-09-16T13:49:02.966363+0000","last_fullsized":"2024-09-16T13:49:02.966363+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:49:02.966079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:49:02.966079+0000","last_clean_scrub_stamp":"2024-09-16T13:49:02.966079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:42:20.670584+0000","scrub_duration":0.0032263169999999998,"objects_trimmed":0,"snaptrim_duration":0.00087664800000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1943,"reported_epoch":1933,"state":"active+clean","last_fresh":"2024-09-16T13:49:08.921637+0000","last_change":"2024-09-16T13:49:08.921578+0000","last_active":"2024-09-16T13:49:08.921637+0000","last_peered":"2024-09-16T13:49:08.921637+0000","last_clean":"2024-09-16T13:49:08.921637+0000","last_became_active":"2024-09-16T12:21:30.718053+0000","last_became_peered":"2024-09-16T12:21:30.718053+0000","last_unstale":"2024-09-16T13:49:08.921637+0000","last_undegraded":"2024-09-16T13:49:08.921637+0000","last_fullsized":"2024-09-16T13:49:08.921637+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:49:08.921335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:49:08.921335+0000","last_clean_scrub_stamp":"2024-09-16T13:49:08.921335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:27:19.341480+0000","scrub_duration":0.0032142730000000001,"objects_trimmed":0,"snaptrim_duration":0.000705296,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1943,"reported_epoch":1933,"state":"active+clean","last_fresh":"2024-09-16T13:49:07.002118+0000","last_change":"2024-09-16T13:49:07.002041+0000","last_active":"2024-09-16T13:49:07.002118+0000","last_peered":"2024-09-16T13:49:07.002118+0000","last_clean":"2024-09-16T13:49:07.002118+0000","last_became_active":"2024-09-16T12:21:30.718170+0000","last_became_peered":"2024-09-16T12:21:30.718170+0000","last_unstale":"2024-09-16T13:49:07.002118+0000","last_undegraded":"2024-09-16T13:49:07.002118+0000","last_fullsized":"2024-09-16T13:49:07.002118+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:49:07.001821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:49:07.001821+0000","last_clean_scrub_stamp":"2024-09-16T13:49:07.001821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:23:30.073420+0000","scrub_duration":0.0031615520000000002,"objects_trimmed":0,"snaptrim_duration":0.0010672889999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1943,"reported_epoch":1933,"state":"active+clean","last_fresh":"2024-09-16T13:49:07.961814+0000","last_change":"2024-09-16T13:49:07.961735+0000","last_active":"2024-09-16T13:49:07.961814+0000","last_peered":"2024-09-16T13:49:07.961814+0000","last_clean":"2024-09-16T13:49:07.961814+0000","last_became_active":"2024-09-16T12:21:30.722079+0000","last_became_peered":"2024-09-16T12:21:30.722079+0000","last_unstale":"2024-09-16T13:49:07.961814+0000","last_undegraded":"2024-09-16T13:49:07.961814+0000","last_fullsized":"2024-09-16T13:49:07.961814+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:49:07.961498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:49:07.961498+0000","last_clean_scrub_stamp":"2024-09-16T13:49:07.961498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:52:05.899759+0000","scrub_duration":0.003113546,"objects_trimmed":0,"snaptrim_duration":0.00051077700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"1933'3","reported_seq":1945,"reported_epoch":1933,"state":"active+clean","last_fresh":"2024-09-16T13:48:52.554554+0000","last_change":"2024-09-16T13:48:52.554495+0000","last_active":"2024-09-16T13:48:52.554554+0000","last_peered":"2024-09-16T13:48:52.554554+0000","last_clean":"2024-09-16T13:48:52.554554+0000","last_became_active":"2024-09-16T12:21:30.721610+0000","last_became_peered":"2024-09-16T12:21:30.721610+0000","last_unstale":"2024-09-16T13:48:52.554554+0000","last_undegraded":"2024-09-16T13:48:52.554554+0000","last_fullsized":"2024-09-16T13:48:52.554554+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"1933'3","last_scrub_stamp":"2024-09-16T13:48:52.554288+0000","last_deep_scrub":"1933'3","last_deep_scrub_stamp":"2024-09-16T13:48:52.554288+0000","last_clean_scrub_stamp":"2024-09-16T13:48:52.554288+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:42:19.324696+0000","scrub_duration":0.0064325509999999999,"objects_trimmed":0,"snaptrim_duration":0.0014060990000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1943,"reported_epoch":1933,"state":"active+clean","last_fresh":"2024-09-16T13:48:53.204630+0000","last_change":"2024-09-16T13:48:53.204500+0000","last_active":"2024-09-16T13:48:53.204630+0000","last_peered":"2024-09-16T13:48:53.204630+0000","last_clean":"2024-09-16T13:48:53.204630+0000","last_became_active":"2024-09-16T12:21:30.724004+0000","last_became_peered":"2024-09-16T12:21:30.724004+0000","last_unstale":"2024-09-16T13:48:53.204630+0000","last_undegraded":"2024-09-16T13:48:53.204630+0000","last_fullsized":"2024-09-16T13:48:53.204630+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:48:53.204127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:48:53.204127+0000","last_clean_scrub_stamp":"2024-09-16T13:48:53.204127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:00:07.224110+0000","scrub_duration":0.0039067379999999999,"objects_trimmed":0,"snaptrim_duration":0.0010506230000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1943,"reported_epoch":1933,"state":"active+clean","last_fresh":"2024-09-16T13:48:49.186785+0000","last_change":"2024-09-16T13:48:49.186746+0000","last_active":"2024-09-16T13:48:49.186785+0000","last_peered":"2024-09-16T13:48:49.186785+0000","last_clean":"2024-09-16T13:48:49.186785+0000","last_became_active":"2024-09-16T12:21:30.721621+0000","last_became_peered":"2024-09-16T12:21:30.721621+0000","last_unstale":"2024-09-16T13:48:49.186785+0000","last_undegraded":"2024-09-16T13:48:49.186785+0000","last_fullsized":"2024-09-16T13:48:49.186785+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:48:49.186576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:48:49.186576+0000","last_clean_scrub_stamp":"2024-09-16T13:48:49.186576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T18:08:50.755168+0000","scrub_duration":0.002865397,"objects_trimmed":0,"snaptrim_duration":0.001027781,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"1933'2","reported_seq":1945,"reported_epoch":1933,"state":"active+clean","last_fresh":"2024-09-16T13:48:59.917536+0000","last_change":"2024-09-16T13:48:59.917442+0000","last_active":"2024-09-16T13:48:59.917536+0000","last_peered":"2024-09-16T13:48:59.917536+0000","last_clean":"2024-09-16T13:48:59.917536+0000","last_became_active":"2024-09-16T12:21:30.717785+0000","last_became_peered":"2024-09-16T12:21:30.717785+0000","last_unstale":"2024-09-16T13:48:59.917536+0000","last_undegraded":"2024-09-16T13:48:59.917536+0000","last_fullsized":"2024-09-16T13:48:59.917536+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"1933'2","last_scrub_stamp":"2024-09-16T13:48:59.917109+0000","last_deep_scrub":"1933'2","last_deep_scrub_stamp":"2024-09-16T13:48:59.917109+0000","last_clean_scrub_stamp":"2024-09-16T13:48:59.917109+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:26:12.885514+0000","scrub_duration":0.0069628520000000003,"objects_trimmed":0,"snaptrim_duration":0.0012262880000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"1933'54","reported_seq":2009,"reported_epoch":1933,"state":"active+clean","last_fresh":"2024-09-16T13:48:50.966566+0000","last_change":"2024-09-16T13:48:50.966487+0000","last_active":"2024-09-16T13:48:50.966566+0000","last_peered":"2024-09-16T13:48:50.966566+0000","last_clean":"2024-09-16T13:48:50.966566+0000","last_became_active":"2024-09-16T12:21:26.696232+0000","last_became_peered":"2024-09-16T12:21:26.696232+0000","last_unstale":"2024-09-16T13:48:50.966566+0000","last_undegraded":"2024-09-16T13:48:50.966566+0000","last_fullsized":"2024-09-16T13:48:50.966566+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"1933'54","last_scrub_stamp":"2024-09-16T13:48:50.966345+0000","last_deep_scrub":"1933'54","last_deep_scrub_stamp":"2024-09-16T13:48:50.966345+0000","last_clean_scrub_stamp":"2024-09-16T13:48:50.966345+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T19:27:41.193137+0000","scrub_duration":0.0090738499999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739430,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914692,"kb_used_data":5628,"kb_used_omap":1,"kb_used_meta":909054,"kb_avail":93457148,"statfs":{"total":96636764160,"available":95700119552,"internally_reserved":0,"allocated":5763072,"data_stored":2799968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":930871640},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:48:24 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":0.66700000000000004,"15min":0.63600000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.184},"max":{"1min":14.237,"5min":14.237,"15min":14.237},"last":0.441},{"interface":"front","average":{"1min":2.0299999999999998,"5min":0.90700000000000003,"15min":0.73999999999999999},"min":{"1min":0.30199999999999999,"5min":0.25,"15min":0.23100000000000001},"max":{"1min":13.818,"5min":13.818,"15min":13.818},"last":0.51500000000000001}]},{"osd":1,"last update":"Mon Sep 16 13:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.61899999999999999,"15min":0.68799999999999994},"min":{"1min":0.26300000000000001,"5min":0.26200000000000001,"15min":0.215},"max":{"1min":2.7570000000000001,"5min":2.7570000000000001,"15min":10.692},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.68000000000000005,"15min":0.749},"min":{"1min":0.27600000000000002,"5min":0.27400000000000002,"15min":0.22900000000000001},"max":{"1min":2.7090000000000001,"5min":2.8380000000000001,"15min":10.747},"last":0.624}]}]},{"osd":1,"up_from":8,"seq":34359739433,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":903180,"kb_used_data":6212,"kb_used_omap":1,"kb_used_meta":896958,"kb_avail":93468660,"statfs":{"total":96636764160,"available":95711907840,"internally_reserved":0,"allocated":6361088,"data_stored":3390355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918485336},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:48:26 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.70099999999999996,"15min":0.68899999999999995},"min":{"1min":0.22,"5min":0.22,"15min":0.182},"max":{"1min":5.641,"5min":5.641,"15min":7.1600000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.65800000000000003,"15min":0.61699999999999999},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.22700000000000001},"max":{"1min":3.5110000000000001,"5min":3.552,"15min":3.585},"last":0.73599999999999999}]},{"osd":2,"last update":"Mon Sep 16 13:48:26 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.63,"15min":0.63100000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":3.4740000000000002,"5min":3.4740000000000002,"15min":4.6950000000000003},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.64800000000000002,"15min":0.67500000000000004},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":3.0350000000000001,"5min":3.0350000000000001,"15min":3.5369999999999999},"last":0.80800000000000005}]}]},{"osd":0,"up_from":8,"seq":34359739430,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":920652,"kb_used_data":6212,"kb_used_omap":1,"kb_used_meta":914430,"kb_avail":93451188,"statfs":{"total":96636764160,"available":95694016512,"internally_reserved":0,"allocated":6361088,"data_stored":3390355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":936376664},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:49:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":1.1000000000000001,"15min":0.79000000000000004},"min":{"1min":0.46000000000000002,"5min":0.22600000000000001,"15min":0.19500000000000001},"max":{"1min":0.67000000000000004,"5min":57.963000000000001,"15min":57.963000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.65100000000000002,"15min":0.61599999999999999},"min":{"1min":0.374,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.76700000000000002,"5min":5.1820000000000004,"15min":5.1820000000000004},"last":0.60599999999999998}]},{"osd":2,"last update":"Mon Sep 16 13:49:10 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":1.0289999999999999,"15min":0.89000000000000001},"min":{"1min":0.45800000000000002,"5min":0.219,"15min":0.219},"max":{"1min":1.2170000000000001,"5min":17.372,"15min":17.372},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.71099999999999997,"15min":0.67800000000000005},"min":{"1min":0.432,"5min":0.254,"15min":0.254},"max":{"1min":0.90900000000000003,"5min":7.3049999999999997,"15min":7.3049999999999997},"last":0.68300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:49:29.808 DEBUG:teuthology.orchestra.run.smithi172:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-09-16T13:49:30.124 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T13:49:30.124 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T13:49:30.124 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-09-16T13:49:30.124 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:49:31.905 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2024-09-16T13:49:36.226 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-09-16T13:49:36.226 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-09-16T13:49:36.227 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:49:37.209 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~0s 2024-09-16T13:49:37.209 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 has been restored 2024-09-16T13:49:42.328 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-09-16T13:49:42.328 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-09-16T13:49:42.329 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:49:42.423 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-09-16T13:49:42.424 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T13:49:42.424 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-09-16T13:49:42.424 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:49:42.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~5s 2024-09-16T13:49:42.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.mgr.x is failed for ~0s 2024-09-16T13:49:42.488 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-09-16T13:49:42.488 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T13:49:42.489 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-09-16T13:49:42.489 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:49:42.517 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-09-16T13:49:42.518 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T13:49:42.518 DEBUG:teuthology.orchestra.run.smithi172:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-16T13:49:42.584 INFO:teuthology.orchestra.run.smithi172.stdout:1726489402.27115 mon.a (mon.0) 210 : cluster [WRN] Health check failed: Degraded data redundancy: 11/1270 objects degraded (0.866%), 1 pg degraded (PG_DEGRADED) 2024-09-16T13:49:42.585 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-09-16T13:49:42.586 DEBUG:teuthology.orchestra.run.smithi172:> sudo egrep '\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-16T13:49:42.603 DEBUG:teuthology.orchestra.run.smithi172:> sudo egrep '\[ERR\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-16T13:49:42.663 DEBUG:teuthology.orchestra.run.smithi172:> sudo egrep '\[WRN\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-16T13:49:42.676 INFO:teuthology.orchestra.run.smithi172.stdout:1726489402.27115 mon.a (mon.0) 210 : cluster [WRN] Health check failed: Degraded data redundancy: 11/1270 objects degraded (0.866%), 1 pg degraded (PG_DEGRADED) 2024-09-16T13:49:42.677 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi172.front.sepia.ceph.com 2024-09-16T13:49:42.677 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-09-16T13:49:42.934 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi172.front.sepia.ceph.com 2024-09-16T13:49:42.934 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-09-16T13:49:43.187 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi172.front.sepia.ceph.com 2024-09-16T13:49:43.187 DEBUG:teuthology.orchestra.run.smithi172:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-09-16T13:49:43.403 INFO:tasks.ceph:Archiving mon data... 2024-09-16T13:49:43.405 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907248/data/cluster2.mon.a.tgz 2024-09-16T13:49:43.405 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2024-09-16T13:49:43.410 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.Diul1oK0pA 2024-09-16T13:49:43.410 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.Diul1oK0pA 2024-09-16T13:49:43.534 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.Diul1oK0pA 2024-09-16T13:49:43.581 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.Diul1oK0pA is 538KB 2024-09-16T13:49:43.646 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.Diul1oK0pA 2024-09-16T13:49:43.653 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T13:49:43.653 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T13:49:43.656 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T13:49:43.839 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-16T13:49:43.850 INFO:tasks.ceph.ceph_manager.cluster1:Canceling any pending splits or merges... 2024-09-16T13:49:43.850 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T13:49:44.275 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T13:49:44.275 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":1928,"fsid":"3d80de80-eeb4-495e-a052-ec9a35b35bbd","created":"2024-09-16T12:20:37.527152+0000","modified":"2024-09-16T13:48:41.003009+0000","last_up_change":"2024-09-16T12:20:46.517261+0000","last_in_change":"2024-09-16T12:20:38.300635+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T12:20:47.990458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T12:20:52.731352+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2f63d01a-c8dc-4d07-bb9e-7866d5784dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":21164},{"type":"v1","addr":"172.21.15.18:6801","nonce":21164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":21164},{"type":"v1","addr":"172.21.15.18:6803","nonce":21164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":21164},{"type":"v1","addr":"172.21.15.18:6807","nonce":21164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":21164},{"type":"v1","addr":"172.21.15.18:6805","nonce":21164}]},"public_addr":"172.21.15.18:6801/21164","cluster_addr":"172.21.15.18:6803/21164","heartbeat_back_addr":"172.21.15.18:6807/21164","heartbeat_front_addr":"172.21.15.18:6805/21164","state":["exists","up"]},{"osd":1,"uuid":"dd97ec5a-4f4a-4a34-82c1-403d5b9ac50b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6816","nonce":21167},{"type":"v1","addr":"172.21.15.18:6817","nonce":21167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6818","nonce":21167},{"type":"v1","addr":"172.21.15.18:6819","nonce":21167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6822","nonce":21167},{"type":"v1","addr":"172.21.15.18:6823","nonce":21167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6820","nonce":21167},{"type":"v1","addr":"172.21.15.18:6821","nonce":21167}]},"public_addr":"172.21.15.18:6817/21167","cluster_addr":"172.21.15.18:6819/21167","heartbeat_back_addr":"172.21.15.18:6823/21167","heartbeat_front_addr":"172.21.15.18:6821/21167","state":["exists","up"]},{"osd":2,"uuid":"0f4108f3-b884-44c1-b94e-e3a5f8f6d4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6808","nonce":21165},{"type":"v1","addr":"172.21.15.18:6809","nonce":21165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6810","nonce":21165},{"type":"v1","addr":"172.21.15.18:6811","nonce":21165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6814","nonce":21165},{"type":"v1","addr":"172.21.15.18:6815","nonce":21165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6812","nonce":21165},{"type":"v1","addr":"172.21.15.18:6813","nonce":21165}]},"public_addr":"172.21.15.18:6809/21165","cluster_addr":"172.21.15.18:6811/21165","heartbeat_back_addr":"172.21.15.18:6815/21165","heartbeat_front_addr":"172.21.15.18:6813/21165","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:20:43.407329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:20:43.768607+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:20:43.654909+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6542,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:49:44.289 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-16T13:49:44.289 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T13:49:44.583 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T13:49:44.586 INFO:teuthology.orchestra.run.smithi018.stderr:dumped all 2024-09-16T13:49:44.597 INFO:teuthology.orchestra.run.smithi018.stdout:{"pg_ready":true,"pg_map":{"version":4594,"stamp":"2024-09-16T13:49:43.965438+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3365396,"kb_used_data":20584,"kb_used_omap":4,"kb_used_meta":3315451,"kb_avail":279750124,"statfs":{"total":289910292480,"available":286464126976,"internally_reserved":0,"allocated":21078016,"data_stored":10486540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":3395021832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002975"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1926,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.242283+0000","last_change":"2024-09-16T12:20:56.574082+0000","last_active":"2024-09-16T13:48:39.242283+0000","last_peered":"2024-09-16T13:48:39.242283+0000","last_clean":"2024-09-16T13:48:39.242283+0000","last_became_active":"2024-09-16T12:20:54.569007+0000","last_became_peered":"2024-09-16T12:20:54.569007+0000","last_unstale":"2024-09-16T13:48:39.242283+0000","last_undegraded":"2024-09-16T13:48:39.242283+0000","last_fullsized":"2024-09-16T13:48:39.242283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:44:06.351794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055482299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1926,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.234488+0000","last_change":"2024-09-16T12:20:56.574146+0000","last_active":"2024-09-16T13:48:39.234488+0000","last_peered":"2024-09-16T13:48:39.234488+0000","last_clean":"2024-09-16T13:48:39.234488+0000","last_became_active":"2024-09-16T12:20:54.571536+0000","last_became_peered":"2024-09-16T12:20:54.571536+0000","last_unstale":"2024-09-16T13:48:39.234488+0000","last_undegraded":"2024-09-16T13:48:39.234488+0000","last_fullsized":"2024-09-16T13:48:39.234488+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:27.211269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062020100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1926,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.241965+0000","last_change":"2024-09-16T12:20:56.575084+0000","last_active":"2024-09-16T13:48:39.241965+0000","last_peered":"2024-09-16T13:48:39.241965+0000","last_clean":"2024-09-16T13:48:39.241965+0000","last_became_active":"2024-09-16T12:20:54.566716+0000","last_became_peered":"2024-09-16T12:20:54.566716+0000","last_unstale":"2024-09-16T13:48:39.241965+0000","last_undegraded":"2024-09-16T13:48:39.241965+0000","last_fullsized":"2024-09-16T13:48:39.241965+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:07.884926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015293030000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1926,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.223794+0000","last_change":"2024-09-16T12:20:56.574539+0000","last_active":"2024-09-16T13:48:39.223794+0000","last_peered":"2024-09-16T13:48:39.223794+0000","last_clean":"2024-09-16T13:48:39.223794+0000","last_became_active":"2024-09-16T12:20:54.570346+0000","last_became_peered":"2024-09-16T12:20:54.570346+0000","last_unstale":"2024-09-16T13:48:39.223794+0000","last_undegraded":"2024-09-16T13:48:39.223794+0000","last_fullsized":"2024-09-16T13:48:39.223794+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:09:55.562224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086539699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1929,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:38.850221+0000","last_change":"2024-09-16T12:20:56.568734+0000","last_active":"2024-09-16T13:48:38.850221+0000","last_peered":"2024-09-16T13:48:38.850221+0000","last_clean":"2024-09-16T13:48:38.850221+0000","last_became_active":"2024-09-16T12:20:54.567480+0000","last_became_peered":"2024-09-16T12:20:54.567480+0000","last_unstale":"2024-09-16T13:48:38.850221+0000","last_undegraded":"2024-09-16T13:48:38.850221+0000","last_fullsized":"2024-09-16T13:48:38.850221+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:06.038597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089212600000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1927,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.037780+0000","last_change":"2024-09-16T12:20:56.647363+0000","last_active":"2024-09-16T13:48:39.037780+0000","last_peered":"2024-09-16T13:48:39.037780+0000","last_clean":"2024-09-16T13:48:39.037780+0000","last_became_active":"2024-09-16T12:20:54.564812+0000","last_became_peered":"2024-09-16T12:20:54.564812+0000","last_unstale":"2024-09-16T13:48:39.037780+0000","last_undegraded":"2024-09-16T13:48:39.037780+0000","last_fullsized":"2024-09-16T13:48:39.037780+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:19:30.492388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093612199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1927,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.030482+0000","last_change":"2024-09-16T12:20:56.647314+0000","last_active":"2024-09-16T13:48:39.030482+0000","last_peered":"2024-09-16T13:48:39.030482+0000","last_clean":"2024-09-16T13:48:39.030482+0000","last_became_active":"2024-09-16T12:20:54.566954+0000","last_became_peered":"2024-09-16T12:20:54.566954+0000","last_unstale":"2024-09-16T13:48:39.030482+0000","last_undegraded":"2024-09-16T13:48:39.030482+0000","last_fullsized":"2024-09-16T13:48:39.030482+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:04:51.826803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086513400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1927,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.247167+0000","last_change":"2024-09-16T12:20:56.574956+0000","last_active":"2024-09-16T13:48:39.247167+0000","last_peered":"2024-09-16T13:48:39.247167+0000","last_clean":"2024-09-16T13:48:39.247167+0000","last_became_active":"2024-09-16T12:20:54.564938+0000","last_became_peered":"2024-09-16T12:20:54.564938+0000","last_unstale":"2024-09-16T13:48:39.247167+0000","last_undegraded":"2024-09-16T13:48:39.247167+0000","last_fullsized":"2024-09-16T13:48:39.247167+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:03:31.051045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011678050000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1991,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.214997+0000","last_change":"2024-09-16T12:20:50.384662+0000","last_active":"2024-09-16T13:48:39.214997+0000","last_peered":"2024-09-16T13:48:39.214997+0000","last_clean":"2024-09-16T13:48:39.214997+0000","last_became_active":"2024-09-16T12:20:50.381536+0000","last_became_peered":"2024-09-16T12:20:50.381536+0000","last_unstale":"2024-09-16T13:48:39.214997+0000","last_undegraded":"2024-09-16T13:48:39.214997+0000","last_fullsized":"2024-09-16T13:48:39.214997+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:48.519944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:48.519944+0000","last_clean_scrub_stamp":"2024-09-16T12:20:48.519944+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:18.620803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739442,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914516,"kb_used_data":6472,"kb_used_omap":1,"kb_used_meta":908030,"kb_avail":93457324,"statfs":{"total":96636764160,"available":95700299776,"internally_reserved":0,"allocated":6627328,"data_stored":3101922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":929823064},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.52500000000000002,"15min":0.52800000000000002},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.69999999999999996,"5min":2.589,"15min":3.4620000000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.68100000000000005,"15min":0.67300000000000004},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.80100000000000005,"5min":2.698,"15min":3.5049999999999999},"last":0.71999999999999997}]},{"osd":1,"last update":"Mon Sep 16 13:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57999999999999996,"15min":0.54600000000000004},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20100000000000001},"max":{"1min":1.006,"5min":3.194,"15min":3.194},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.73699999999999999,"15min":0.69899999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.23599999999999999},"max":{"1min":1.1439999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":0.66900000000000004}]}]},{"osd":1,"up_from":8,"seq":34359739441,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1238180,"kb_used_data":7056,"kb_used_omap":1,"kb_used_meta":1201790,"kb_avail":93133660,"statfs":{"total":96636764160,"available":95368867840,"internally_reserved":0,"allocated":7225344,"data_stored":3692309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1230633304},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.66400000000000003,"15min":0.59699999999999998},"min":{"1min":0.40400000000000003,"5min":0.32500000000000001,"15min":0.188},"max":{"1min":4.8920000000000003,"5min":4.8920000000000003,"15min":4.8920000000000003},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.64100000000000001,"15min":0.66700000000000004},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":4.641},"last":0.54100000000000004}]},{"osd":2,"last update":"Mon Sep 16 13:49:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.69699999999999995,"15min":0.67900000000000005},"min":{"1min":0.44800000000000001,"5min":0.23200000000000001,"15min":0.218},"max":{"1min":3.629,"5min":4.8479999999999999,"15min":4.8479999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.78300000000000003,"15min":0.71199999999999997},"min":{"1min":0.61399999999999999,"5min":0.27000000000000002,"15min":0.253},"max":{"1min":3.6669999999999998,"5min":4.9359999999999999,"15min":4.9359999999999999},"last":0.66300000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739443,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1212700,"kb_used_data":7056,"kb_used_omap":1,"kb_used_meta":1205630,"kb_avail":93159140,"statfs":{"total":96636764160,"available":95394959360,"internally_reserved":0,"allocated":7225344,"data_stored":3692309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1234565464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:49:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.52600000000000002,"15min":0.54100000000000004},"min":{"1min":0.317,"5min":0.20899999999999999,"15min":0.20100000000000001},"max":{"1min":0.74299999999999999,"5min":1.1060000000000001,"15min":3.2069999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66300000000000003,"15min":0.747},"min":{"1min":0.40300000000000002,"5min":0.29199999999999998,"15min":0.222},"max":{"1min":0.81999999999999995,"5min":2.4209999999999998,"15min":10.430999999999999},"last":0.71599999999999997}]},{"osd":2,"last update":"Mon Sep 16 13:49:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.23499999999999999,"15min":0.215},"max":{"1min":0.66500000000000004,"5min":1.6479999999999999,"15min":4.3090000000000002},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.70399999999999996,"15min":0.67900000000000005},"min":{"1min":0.42799999999999999,"5min":0.31,"15min":0.25},"max":{"1min":0.86499999999999999,"5min":1.6879999999999999,"15min":4.3869999999999996},"last":0.755}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:49:44.598 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T13:49:44.887 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T13:49:44.889 INFO:teuthology.orchestra.run.smithi018.stderr:dumped all 2024-09-16T13:49:44.899 INFO:teuthology.orchestra.run.smithi018.stdout:{"pg_ready":true,"pg_map":{"version":4594,"stamp":"2024-09-16T13:49:43.965438+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3365396,"kb_used_data":20584,"kb_used_omap":4,"kb_used_meta":3315451,"kb_avail":279750124,"statfs":{"total":289910292480,"available":286464126976,"internally_reserved":0,"allocated":21078016,"data_stored":10486540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":3395021832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002975"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1926,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.242283+0000","last_change":"2024-09-16T12:20:56.574082+0000","last_active":"2024-09-16T13:48:39.242283+0000","last_peered":"2024-09-16T13:48:39.242283+0000","last_clean":"2024-09-16T13:48:39.242283+0000","last_became_active":"2024-09-16T12:20:54.569007+0000","last_became_peered":"2024-09-16T12:20:54.569007+0000","last_unstale":"2024-09-16T13:48:39.242283+0000","last_undegraded":"2024-09-16T13:48:39.242283+0000","last_fullsized":"2024-09-16T13:48:39.242283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:44:06.351794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055482299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1926,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.234488+0000","last_change":"2024-09-16T12:20:56.574146+0000","last_active":"2024-09-16T13:48:39.234488+0000","last_peered":"2024-09-16T13:48:39.234488+0000","last_clean":"2024-09-16T13:48:39.234488+0000","last_became_active":"2024-09-16T12:20:54.571536+0000","last_became_peered":"2024-09-16T12:20:54.571536+0000","last_unstale":"2024-09-16T13:48:39.234488+0000","last_undegraded":"2024-09-16T13:48:39.234488+0000","last_fullsized":"2024-09-16T13:48:39.234488+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:27.211269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062020100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1926,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.241965+0000","last_change":"2024-09-16T12:20:56.575084+0000","last_active":"2024-09-16T13:48:39.241965+0000","last_peered":"2024-09-16T13:48:39.241965+0000","last_clean":"2024-09-16T13:48:39.241965+0000","last_became_active":"2024-09-16T12:20:54.566716+0000","last_became_peered":"2024-09-16T12:20:54.566716+0000","last_unstale":"2024-09-16T13:48:39.241965+0000","last_undegraded":"2024-09-16T13:48:39.241965+0000","last_fullsized":"2024-09-16T13:48:39.241965+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:07.884926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015293030000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1926,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.223794+0000","last_change":"2024-09-16T12:20:56.574539+0000","last_active":"2024-09-16T13:48:39.223794+0000","last_peered":"2024-09-16T13:48:39.223794+0000","last_clean":"2024-09-16T13:48:39.223794+0000","last_became_active":"2024-09-16T12:20:54.570346+0000","last_became_peered":"2024-09-16T12:20:54.570346+0000","last_unstale":"2024-09-16T13:48:39.223794+0000","last_undegraded":"2024-09-16T13:48:39.223794+0000","last_fullsized":"2024-09-16T13:48:39.223794+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:09:55.562224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086539699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1929,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:38.850221+0000","last_change":"2024-09-16T12:20:56.568734+0000","last_active":"2024-09-16T13:48:38.850221+0000","last_peered":"2024-09-16T13:48:38.850221+0000","last_clean":"2024-09-16T13:48:38.850221+0000","last_became_active":"2024-09-16T12:20:54.567480+0000","last_became_peered":"2024-09-16T12:20:54.567480+0000","last_unstale":"2024-09-16T13:48:38.850221+0000","last_undegraded":"2024-09-16T13:48:38.850221+0000","last_fullsized":"2024-09-16T13:48:38.850221+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:06.038597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089212600000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1927,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.037780+0000","last_change":"2024-09-16T12:20:56.647363+0000","last_active":"2024-09-16T13:48:39.037780+0000","last_peered":"2024-09-16T13:48:39.037780+0000","last_clean":"2024-09-16T13:48:39.037780+0000","last_became_active":"2024-09-16T12:20:54.564812+0000","last_became_peered":"2024-09-16T12:20:54.564812+0000","last_unstale":"2024-09-16T13:48:39.037780+0000","last_undegraded":"2024-09-16T13:48:39.037780+0000","last_fullsized":"2024-09-16T13:48:39.037780+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:19:30.492388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093612199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1927,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.030482+0000","last_change":"2024-09-16T12:20:56.647314+0000","last_active":"2024-09-16T13:48:39.030482+0000","last_peered":"2024-09-16T13:48:39.030482+0000","last_clean":"2024-09-16T13:48:39.030482+0000","last_became_active":"2024-09-16T12:20:54.566954+0000","last_became_peered":"2024-09-16T12:20:54.566954+0000","last_unstale":"2024-09-16T13:48:39.030482+0000","last_undegraded":"2024-09-16T13:48:39.030482+0000","last_fullsized":"2024-09-16T13:48:39.030482+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:04:51.826803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086513400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1927,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.247167+0000","last_change":"2024-09-16T12:20:56.574956+0000","last_active":"2024-09-16T13:48:39.247167+0000","last_peered":"2024-09-16T13:48:39.247167+0000","last_clean":"2024-09-16T13:48:39.247167+0000","last_became_active":"2024-09-16T12:20:54.564938+0000","last_became_peered":"2024-09-16T12:20:54.564938+0000","last_unstale":"2024-09-16T13:48:39.247167+0000","last_undegraded":"2024-09-16T13:48:39.247167+0000","last_fullsized":"2024-09-16T13:48:39.247167+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:03:31.051045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011678050000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1991,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.214997+0000","last_change":"2024-09-16T12:20:50.384662+0000","last_active":"2024-09-16T13:48:39.214997+0000","last_peered":"2024-09-16T13:48:39.214997+0000","last_clean":"2024-09-16T13:48:39.214997+0000","last_became_active":"2024-09-16T12:20:50.381536+0000","last_became_peered":"2024-09-16T12:20:50.381536+0000","last_unstale":"2024-09-16T13:48:39.214997+0000","last_undegraded":"2024-09-16T13:48:39.214997+0000","last_fullsized":"2024-09-16T13:48:39.214997+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:48.519944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:48.519944+0000","last_clean_scrub_stamp":"2024-09-16T12:20:48.519944+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:18.620803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739442,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914516,"kb_used_data":6472,"kb_used_omap":1,"kb_used_meta":908030,"kb_avail":93457324,"statfs":{"total":96636764160,"available":95700299776,"internally_reserved":0,"allocated":6627328,"data_stored":3101922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":929823064},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.52500000000000002,"15min":0.52800000000000002},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.69999999999999996,"5min":2.589,"15min":3.4620000000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.68100000000000005,"15min":0.67300000000000004},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.80100000000000005,"5min":2.698,"15min":3.5049999999999999},"last":0.71999999999999997}]},{"osd":1,"last update":"Mon Sep 16 13:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57999999999999996,"15min":0.54600000000000004},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20100000000000001},"max":{"1min":1.006,"5min":3.194,"15min":3.194},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.73699999999999999,"15min":0.69899999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.23599999999999999},"max":{"1min":1.1439999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":0.66900000000000004}]}]},{"osd":1,"up_from":8,"seq":34359739441,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1238180,"kb_used_data":7056,"kb_used_omap":1,"kb_used_meta":1201790,"kb_avail":93133660,"statfs":{"total":96636764160,"available":95368867840,"internally_reserved":0,"allocated":7225344,"data_stored":3692309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1230633304},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.66400000000000003,"15min":0.59699999999999998},"min":{"1min":0.40400000000000003,"5min":0.32500000000000001,"15min":0.188},"max":{"1min":4.8920000000000003,"5min":4.8920000000000003,"15min":4.8920000000000003},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.64100000000000001,"15min":0.66700000000000004},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":4.641},"last":0.54100000000000004}]},{"osd":2,"last update":"Mon Sep 16 13:49:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.69699999999999995,"15min":0.67900000000000005},"min":{"1min":0.44800000000000001,"5min":0.23200000000000001,"15min":0.218},"max":{"1min":3.629,"5min":4.8479999999999999,"15min":4.8479999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.78300000000000003,"15min":0.71199999999999997},"min":{"1min":0.61399999999999999,"5min":0.27000000000000002,"15min":0.253},"max":{"1min":3.6669999999999998,"5min":4.9359999999999999,"15min":4.9359999999999999},"last":0.66300000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739443,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1212700,"kb_used_data":7056,"kb_used_omap":1,"kb_used_meta":1205630,"kb_avail":93159140,"statfs":{"total":96636764160,"available":95394959360,"internally_reserved":0,"allocated":7225344,"data_stored":3692309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1234565464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:49:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.52600000000000002,"15min":0.54100000000000004},"min":{"1min":0.317,"5min":0.20899999999999999,"15min":0.20100000000000001},"max":{"1min":0.74299999999999999,"5min":1.1060000000000001,"15min":3.2069999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66300000000000003,"15min":0.747},"min":{"1min":0.40300000000000002,"5min":0.29199999999999998,"15min":0.222},"max":{"1min":0.81999999999999995,"5min":2.4209999999999998,"15min":10.430999999999999},"last":0.71599999999999997}]},{"osd":2,"last update":"Mon Sep 16 13:49:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.23499999999999999,"15min":0.215},"max":{"1min":0.66500000000000004,"5min":1.6479999999999999,"15min":4.3090000000000002},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.70399999999999996,"15min":0.67900000000000005},"min":{"1min":0.42799999999999999,"5min":0.31,"15min":0.25},"max":{"1min":0.86499999999999999,"5min":1.6879999999999999,"15min":4.3869999999999996},"last":0.755}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:49:44.900 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-16T13:49:44.901 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T13:49:45.188 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T13:49:45.190 INFO:teuthology.orchestra.run.smithi018.stderr:dumped all 2024-09-16T13:49:45.206 INFO:teuthology.orchestra.run.smithi018.stdout:{"pg_ready":true,"pg_map":{"version":4594,"stamp":"2024-09-16T13:49:43.965438+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3365396,"kb_used_data":20584,"kb_used_omap":4,"kb_used_meta":3315451,"kb_avail":279750124,"statfs":{"total":289910292480,"available":286464126976,"internally_reserved":0,"allocated":21078016,"data_stored":10486540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":3395021832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002975"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1926,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.242283+0000","last_change":"2024-09-16T12:20:56.574082+0000","last_active":"2024-09-16T13:48:39.242283+0000","last_peered":"2024-09-16T13:48:39.242283+0000","last_clean":"2024-09-16T13:48:39.242283+0000","last_became_active":"2024-09-16T12:20:54.569007+0000","last_became_peered":"2024-09-16T12:20:54.569007+0000","last_unstale":"2024-09-16T13:48:39.242283+0000","last_undegraded":"2024-09-16T13:48:39.242283+0000","last_fullsized":"2024-09-16T13:48:39.242283+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:44:06.351794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055482299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1926,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.234488+0000","last_change":"2024-09-16T12:20:56.574146+0000","last_active":"2024-09-16T13:48:39.234488+0000","last_peered":"2024-09-16T13:48:39.234488+0000","last_clean":"2024-09-16T13:48:39.234488+0000","last_became_active":"2024-09-16T12:20:54.571536+0000","last_became_peered":"2024-09-16T12:20:54.571536+0000","last_unstale":"2024-09-16T13:48:39.234488+0000","last_undegraded":"2024-09-16T13:48:39.234488+0000","last_fullsized":"2024-09-16T13:48:39.234488+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T12:41:27.211269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062020100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1926,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.241965+0000","last_change":"2024-09-16T12:20:56.575084+0000","last_active":"2024-09-16T13:48:39.241965+0000","last_peered":"2024-09-16T13:48:39.241965+0000","last_clean":"2024-09-16T13:48:39.241965+0000","last_became_active":"2024-09-16T12:20:54.566716+0000","last_became_peered":"2024-09-16T12:20:54.566716+0000","last_unstale":"2024-09-16T13:48:39.241965+0000","last_undegraded":"2024-09-16T13:48:39.241965+0000","last_fullsized":"2024-09-16T13:48:39.241965+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:01:07.884926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015293030000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1926,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.223794+0000","last_change":"2024-09-16T12:20:56.574539+0000","last_active":"2024-09-16T13:48:39.223794+0000","last_peered":"2024-09-16T13:48:39.223794+0000","last_clean":"2024-09-16T13:48:39.223794+0000","last_became_active":"2024-09-16T12:20:54.570346+0000","last_became_peered":"2024-09-16T12:20:54.570346+0000","last_unstale":"2024-09-16T13:48:39.223794+0000","last_undegraded":"2024-09-16T13:48:39.223794+0000","last_fullsized":"2024-09-16T13:48:39.223794+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T00:09:55.562224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086539699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1929,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:38.850221+0000","last_change":"2024-09-16T12:20:56.568734+0000","last_active":"2024-09-16T13:48:38.850221+0000","last_peered":"2024-09-16T13:48:38.850221+0000","last_clean":"2024-09-16T13:48:38.850221+0000","last_became_active":"2024-09-16T12:20:54.567480+0000","last_became_peered":"2024-09-16T12:20:54.567480+0000","last_unstale":"2024-09-16T13:48:38.850221+0000","last_undegraded":"2024-09-16T13:48:38.850221+0000","last_fullsized":"2024-09-16T13:48:38.850221+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:30:06.038597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089212600000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1927,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.037780+0000","last_change":"2024-09-16T12:20:56.647363+0000","last_active":"2024-09-16T13:48:39.037780+0000","last_peered":"2024-09-16T13:48:39.037780+0000","last_clean":"2024-09-16T13:48:39.037780+0000","last_became_active":"2024-09-16T12:20:54.564812+0000","last_became_peered":"2024-09-16T12:20:54.564812+0000","last_unstale":"2024-09-16T13:48:39.037780+0000","last_undegraded":"2024-09-16T13:48:39.037780+0000","last_fullsized":"2024-09-16T13:48:39.037780+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:19:30.492388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093612199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1927,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.030482+0000","last_change":"2024-09-16T12:20:56.647314+0000","last_active":"2024-09-16T13:48:39.030482+0000","last_peered":"2024-09-16T13:48:39.030482+0000","last_clean":"2024-09-16T13:48:39.030482+0000","last_became_active":"2024-09-16T12:20:54.566954+0000","last_became_peered":"2024-09-16T12:20:54.566954+0000","last_unstale":"2024-09-16T13:48:39.030482+0000","last_undegraded":"2024-09-16T13:48:39.030482+0000","last_fullsized":"2024-09-16T13:48:39.030482+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:04:51.826803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086513400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1927,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.247167+0000","last_change":"2024-09-16T12:20:56.574956+0000","last_active":"2024-09-16T13:48:39.247167+0000","last_peered":"2024-09-16T13:48:39.247167+0000","last_clean":"2024-09-16T13:48:39.247167+0000","last_became_active":"2024-09-16T12:20:54.564938+0000","last_became_peered":"2024-09-16T12:20:54.564938+0000","last_unstale":"2024-09-16T13:48:39.247167+0000","last_undegraded":"2024-09-16T13:48:39.247167+0000","last_fullsized":"2024-09-16T13:48:39.247167+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:03:31.051045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011678050000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"12'52","reported_seq":1991,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.214997+0000","last_change":"2024-09-16T12:20:50.384662+0000","last_active":"2024-09-16T13:48:39.214997+0000","last_peered":"2024-09-16T13:48:39.214997+0000","last_clean":"2024-09-16T13:48:39.214997+0000","last_became_active":"2024-09-16T12:20:50.381536+0000","last_became_peered":"2024-09-16T12:20:50.381536+0000","last_unstale":"2024-09-16T13:48:39.214997+0000","last_undegraded":"2024-09-16T13:48:39.214997+0000","last_fullsized":"2024-09-16T13:48:39.214997+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:48.519944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:48.519944+0000","last_clean_scrub_stamp":"2024-09-16T12:20:48.519944+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T15:12:18.620803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739442,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914516,"kb_used_data":6472,"kb_used_omap":1,"kb_used_meta":908030,"kb_avail":93457324,"statfs":{"total":96636764160,"available":95700299776,"internally_reserved":0,"allocated":6627328,"data_stored":3101922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":929823064},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.52500000000000002,"15min":0.52800000000000002},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.69999999999999996,"5min":2.589,"15min":3.4620000000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.68100000000000005,"15min":0.67300000000000004},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.80100000000000005,"5min":2.698,"15min":3.5049999999999999},"last":0.71999999999999997}]},{"osd":1,"last update":"Mon Sep 16 13:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57999999999999996,"15min":0.54600000000000004},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20100000000000001},"max":{"1min":1.006,"5min":3.194,"15min":3.194},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.73699999999999999,"15min":0.69899999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.23599999999999999},"max":{"1min":1.1439999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":0.66900000000000004}]}]},{"osd":1,"up_from":8,"seq":34359739441,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1238180,"kb_used_data":7056,"kb_used_omap":1,"kb_used_meta":1201790,"kb_avail":93133660,"statfs":{"total":96636764160,"available":95368867840,"internally_reserved":0,"allocated":7225344,"data_stored":3692309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1230633304},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.66400000000000003,"15min":0.59699999999999998},"min":{"1min":0.40400000000000003,"5min":0.32500000000000001,"15min":0.188},"max":{"1min":4.8920000000000003,"5min":4.8920000000000003,"15min":4.8920000000000003},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.64100000000000001,"15min":0.66700000000000004},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":4.641},"last":0.54100000000000004}]},{"osd":2,"last update":"Mon Sep 16 13:49:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.69699999999999995,"15min":0.67900000000000005},"min":{"1min":0.44800000000000001,"5min":0.23200000000000001,"15min":0.218},"max":{"1min":3.629,"5min":4.8479999999999999,"15min":4.8479999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.78300000000000003,"15min":0.71199999999999997},"min":{"1min":0.61399999999999999,"5min":0.27000000000000002,"15min":0.253},"max":{"1min":3.6669999999999998,"5min":4.9359999999999999,"15min":4.9359999999999999},"last":0.66300000000000003}]}]},{"osd":0,"up_from":8,"seq":34359739443,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1212700,"kb_used_data":7056,"kb_used_omap":1,"kb_used_meta":1205630,"kb_avail":93159140,"statfs":{"total":96636764160,"available":95394959360,"internally_reserved":0,"allocated":7225344,"data_stored":3692309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1234565464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:49:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.52600000000000002,"15min":0.54100000000000004},"min":{"1min":0.317,"5min":0.20899999999999999,"15min":0.20100000000000001},"max":{"1min":0.74299999999999999,"5min":1.1060000000000001,"15min":3.2069999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66300000000000003,"15min":0.747},"min":{"1min":0.40300000000000002,"5min":0.29199999999999998,"15min":0.222},"max":{"1min":0.81999999999999995,"5min":2.4209999999999998,"15min":10.430999999999999},"last":0.71599999999999997}]},{"osd":2,"last update":"Mon Sep 16 13:49:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.23499999999999999,"15min":0.215},"max":{"1min":0.66500000000000004,"5min":1.6479999999999999,"15min":4.3090000000000002},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.70399999999999996,"15min":0.67900000000000005},"min":{"1min":0.42799999999999999,"5min":0.31,"15min":0.25},"max":{"1min":0.86499999999999999,"5min":1.6879999999999999,"15min":4.3869999999999996},"last":0.755}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:49:45.207 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-16T13:49:45.501 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T13:49:45.501 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":1928,"fsid":"3d80de80-eeb4-495e-a052-ec9a35b35bbd","created":"2024-09-16T12:20:37.527152+0000","modified":"2024-09-16T13:48:41.003009+0000","last_up_change":"2024-09-16T12:20:46.517261+0000","last_in_change":"2024-09-16T12:20:38.300635+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-16T12:20:47.990458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-16T12:20:52.731352+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":16,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2f63d01a-c8dc-4d07-bb9e-7866d5784dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6800","nonce":21164},{"type":"v1","addr":"172.21.15.18:6801","nonce":21164}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6802","nonce":21164},{"type":"v1","addr":"172.21.15.18:6803","nonce":21164}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6806","nonce":21164},{"type":"v1","addr":"172.21.15.18:6807","nonce":21164}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6804","nonce":21164},{"type":"v1","addr":"172.21.15.18:6805","nonce":21164}]},"public_addr":"172.21.15.18:6801/21164","cluster_addr":"172.21.15.18:6803/21164","heartbeat_back_addr":"172.21.15.18:6807/21164","heartbeat_front_addr":"172.21.15.18:6805/21164","state":["exists","up"]},{"osd":1,"uuid":"dd97ec5a-4f4a-4a34-82c1-403d5b9ac50b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6816","nonce":21167},{"type":"v1","addr":"172.21.15.18:6817","nonce":21167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6818","nonce":21167},{"type":"v1","addr":"172.21.15.18:6819","nonce":21167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6822","nonce":21167},{"type":"v1","addr":"172.21.15.18:6823","nonce":21167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6820","nonce":21167},{"type":"v1","addr":"172.21.15.18:6821","nonce":21167}]},"public_addr":"172.21.15.18:6817/21167","cluster_addr":"172.21.15.18:6819/21167","heartbeat_back_addr":"172.21.15.18:6823/21167","heartbeat_front_addr":"172.21.15.18:6821/21167","state":["exists","up"]},{"osd":2,"uuid":"0f4108f3-b884-44c1-b94e-e3a5f8f6d4d3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6808","nonce":21165},{"type":"v1","addr":"172.21.15.18:6809","nonce":21165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6810","nonce":21165},{"type":"v1","addr":"172.21.15.18:6811","nonce":21165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6814","nonce":21165},{"type":"v1","addr":"172.21.15.18:6815","nonce":21165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.18:6812","nonce":21165},{"type":"v1","addr":"172.21.15.18:6813","nonce":21165}]},"public_addr":"172.21.15.18:6809/21165","cluster_addr":"172.21.15.18:6811/21165","heartbeat_back_addr":"172.21.15.18:6815/21165","heartbeat_front_addr":"172.21.15.18:6813/21165","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:20:43.407329+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:20:43.768607+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-16T12:20:43.654909+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":6542,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-16T13:49:46.517 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-09-16T13:49:46.517 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:49:46.672 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2024-09-16T13:49:46.673 INFO:teuthology.orchestra.run.smithi018.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:49:46.673 INFO:teuthology.orchestra.run.smithi018.stdout:} 2024-09-16T13:49:46.686 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-09-16T13:49:46.990 INFO:teuthology.orchestra.run.smithi018.stderr:instructed osd(s) 0 to deep-scrub 2024-09-16T13:49:47.005 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-09-16T13:49:47.006 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:49:47.154 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2024-09-16T13:49:47.154 INFO:teuthology.orchestra.run.smithi018.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:49:47.154 INFO:teuthology.orchestra.run.smithi018.stdout:} 2024-09-16T13:49:47.166 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-09-16T13:49:47.461 INFO:teuthology.orchestra.run.smithi018.stderr:instructed osd(s) 1 to deep-scrub 2024-09-16T13:49:47.471 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-09-16T13:49:47.471 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-16T13:49:47.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 has been restored 2024-09-16T13:49:47.489 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.mgr.x has been restored 2024-09-16T13:49:47.617 INFO:teuthology.orchestra.run.smithi018.stdout:{ 2024-09-16T13:49:47.617 INFO:teuthology.orchestra.run.smithi018.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-16T13:49:47.617 INFO:teuthology.orchestra.run.smithi018.stdout:} 2024-09-16T13:49:47.633 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-09-16T13:49:47.938 INFO:teuthology.orchestra.run.smithi018.stderr:instructed osd(s) 2 to deep-scrub 2024-09-16T13:49:47.949 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T13:49:48.272 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T13:49:48.277 INFO:teuthology.orchestra.run.smithi018.stderr:dumped all 2024-09-16T13:49:48.289 INFO:teuthology.orchestra.run.smithi018.stdout:{"pg_ready":true,"pg_map":{"version":4596,"stamp":"2024-09-16T13:49:47.966450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":57,"ondisk_log_size":57,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3365396,"kb_used_data":20584,"kb_used_omap":4,"kb_used_meta":3315451,"kb_avail":279750124,"statfs":{"total":289910292480,"available":286464126976,"internally_reserved":0,"allocated":21078016,"data_stored":10486540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":3395021832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002905"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1927,"reported_epoch":1928,"state":"active+clean","last_fresh":"2024-09-16T13:48:41.024637+0000","last_change":"2024-09-16T12:20:56.574082+0000","last_active":"2024-09-16T13:48:41.024637+0000","last_peered":"2024-09-16T13:48:41.024637+0000","last_clean":"2024-09-16T13:48:41.024637+0000","last_became_active":"2024-09-16T12:20:54.569007+0000","last_became_peered":"2024-09-16T12:20:54.569007+0000","last_unstale":"2024-09-16T13:48:41.024637+0000","last_undegraded":"2024-09-16T13:48:41.024637+0000","last_fullsized":"2024-09-16T13:48:41.024637+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055482299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1927,"reported_epoch":1928,"state":"active+clean","last_fresh":"2024-09-16T13:48:41.024151+0000","last_change":"2024-09-16T12:20:56.574146+0000","last_active":"2024-09-16T13:48:41.024151+0000","last_peered":"2024-09-16T13:48:41.024151+0000","last_clean":"2024-09-16T13:48:41.024151+0000","last_became_active":"2024-09-16T12:20:54.571536+0000","last_became_peered":"2024-09-16T12:20:54.571536+0000","last_unstale":"2024-09-16T13:48:41.024151+0000","last_undegraded":"2024-09-16T13:48:41.024151+0000","last_fullsized":"2024-09-16T13:48:41.024151+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062020100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1927,"reported_epoch":1928,"state":"active+clean","last_fresh":"2024-09-16T13:48:41.022042+0000","last_change":"2024-09-16T12:20:56.575084+0000","last_active":"2024-09-16T13:48:41.022042+0000","last_peered":"2024-09-16T13:48:41.022042+0000","last_clean":"2024-09-16T13:48:41.022042+0000","last_became_active":"2024-09-16T12:20:54.566716+0000","last_became_peered":"2024-09-16T12:20:54.566716+0000","last_unstale":"2024-09-16T13:48:41.022042+0000","last_undegraded":"2024-09-16T13:48:41.022042+0000","last_fullsized":"2024-09-16T13:48:41.022042+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015293030000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1927,"reported_epoch":1928,"state":"active+clean","last_fresh":"2024-09-16T13:48:41.018673+0000","last_change":"2024-09-16T12:20:56.574539+0000","last_active":"2024-09-16T13:48:41.018673+0000","last_peered":"2024-09-16T13:48:41.018673+0000","last_clean":"2024-09-16T13:48:41.018673+0000","last_became_active":"2024-09-16T12:20:54.570346+0000","last_became_peered":"2024-09-16T12:20:54.570346+0000","last_unstale":"2024-09-16T13:48:41.018673+0000","last_undegraded":"2024-09-16T13:48:41.018673+0000","last_fullsized":"2024-09-16T13:48:41.018673+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086539699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"16'2","reported_seq":1930,"reported_epoch":1928,"state":"active+clean","last_fresh":"2024-09-16T13:48:42.994821+0000","last_change":"2024-09-16T12:20:56.568734+0000","last_active":"2024-09-16T13:48:42.994821+0000","last_peered":"2024-09-16T13:48:42.994821+0000","last_clean":"2024-09-16T13:48:42.994821+0000","last_became_active":"2024-09-16T12:20:54.567480+0000","last_became_peered":"2024-09-16T12:20:54.567480+0000","last_unstale":"2024-09-16T13:48:42.994821+0000","last_undegraded":"2024-09-16T13:48:42.994821+0000","last_fullsized":"2024-09-16T13:48:42.994821+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089212600000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1927,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.037780+0000","last_change":"2024-09-16T12:20:56.647363+0000","last_active":"2024-09-16T13:48:39.037780+0000","last_peered":"2024-09-16T13:48:39.037780+0000","last_clean":"2024-09-16T13:48:39.037780+0000","last_became_active":"2024-09-16T12:20:54.564812+0000","last_became_peered":"2024-09-16T12:20:54.564812+0000","last_unstale":"2024-09-16T13:48:39.037780+0000","last_undegraded":"2024-09-16T13:48:39.037780+0000","last_fullsized":"2024-09-16T13:48:39.037780+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T13:19:30.492388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093612199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1927,"reported_epoch":1927,"state":"active+clean","last_fresh":"2024-09-16T13:48:39.030482+0000","last_change":"2024-09-16T12:20:56.647314+0000","last_active":"2024-09-16T13:48:39.030482+0000","last_peered":"2024-09-16T13:48:39.030482+0000","last_clean":"2024-09-16T13:48:39.030482+0000","last_became_active":"2024-09-16T12:20:54.566954+0000","last_became_peered":"2024-09-16T12:20:54.566954+0000","last_unstale":"2024-09-16T13:48:39.030482+0000","last_undegraded":"2024-09-16T13:48:39.030482+0000","last_fullsized":"2024-09-16T13:48:39.030482+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:04:51.826803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086513400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"14'1","reported_seq":1928,"reported_epoch":1928,"state":"active+clean","last_fresh":"2024-09-16T13:48:41.024022+0000","last_change":"2024-09-16T12:20:56.574956+0000","last_active":"2024-09-16T13:48:41.024022+0000","last_peered":"2024-09-16T13:48:41.024022+0000","last_clean":"2024-09-16T13:48:41.024022+0000","last_became_active":"2024-09-16T12:20:54.564938+0000","last_became_peered":"2024-09-16T12:20:54.564938+0000","last_unstale":"2024-09-16T13:48:41.024022+0000","last_undegraded":"2024-09-16T13:48:41.024022+0000","last_fullsized":"2024-09-16T13:48:41.024022+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T12:20:53.541416+0000","last_clean_scrub_stamp":"2024-09-16T12:20:53.541416+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011678050000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"1928'54","reported_seq":2004,"reported_epoch":1928,"state":"active+clean","last_fresh":"2024-09-16T13:49:47.571633+0000","last_change":"2024-09-16T13:49:47.571514+0000","last_active":"2024-09-16T13:49:47.571633+0000","last_peered":"2024-09-16T13:49:47.571633+0000","last_clean":"2024-09-16T13:49:47.571633+0000","last_became_active":"2024-09-16T12:20:50.381536+0000","last_became_peered":"2024-09-16T12:20:50.381536+0000","last_unstale":"2024-09-16T13:49:47.571633+0000","last_undegraded":"2024-09-16T13:49:47.571633+0000","last_fullsized":"2024-09-16T13:49:47.571633+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"1928'54","last_scrub_stamp":"2024-09-16T13:49:47.571252+0000","last_deep_scrub":"1928'54","last_deep_scrub_stamp":"2024-09-16T13:49:47.571252+0000","last_clean_scrub_stamp":"2024-09-16T13:49:47.571252+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:43:56.513644+0000","scrub_duration":0.0061503119999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739443,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914516,"kb_used_data":6472,"kb_used_omap":1,"kb_used_meta":908030,"kb_avail":93457324,"statfs":{"total":96636764160,"available":95700299776,"internally_reserved":0,"allocated":6627328,"data_stored":3101922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":929823064},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.52500000000000002,"15min":0.52800000000000002},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.69999999999999996,"5min":2.589,"15min":3.4620000000000002},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.68100000000000005,"15min":0.67300000000000004},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.80100000000000005,"5min":2.698,"15min":3.5049999999999999},"last":0.65600000000000003}]},{"osd":1,"last update":"Mon Sep 16 13:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57999999999999996,"15min":0.54600000000000004},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.20100000000000001},"max":{"1min":1.006,"5min":3.194,"15min":3.194},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.73699999999999999,"15min":0.69899999999999995},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.23599999999999999},"max":{"1min":1.1439999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":0.69899999999999995}]}]},{"osd":1,"up_from":8,"seq":34359739442,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1238180,"kb_used_data":7056,"kb_used_omap":1,"kb_used_meta":1201790,"kb_avail":93133660,"statfs":{"total":96636764160,"available":95368867840,"internally_reserved":0,"allocated":7225344,"data_stored":3692309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1230633304},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.66400000000000003,"15min":0.59699999999999998},"min":{"1min":0.40400000000000003,"5min":0.32500000000000001,"15min":0.188},"max":{"1min":4.8920000000000003,"5min":4.8920000000000003,"15min":4.8920000000000003},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.64100000000000001,"15min":0.66700000000000004},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":4.641},"last":0.56999999999999995}]},{"osd":2,"last update":"Mon Sep 16 13:49:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.69699999999999995,"15min":0.67900000000000005},"min":{"1min":0.44800000000000001,"5min":0.23200000000000001,"15min":0.218},"max":{"1min":3.629,"5min":4.8479999999999999,"15min":4.8479999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.78300000000000003,"15min":0.71199999999999997},"min":{"1min":0.61399999999999999,"5min":0.27000000000000002,"15min":0.253},"max":{"1min":3.6669999999999998,"5min":4.9359999999999999,"15min":4.9359999999999999},"last":0.501}]}]},{"osd":0,"up_from":8,"seq":34359739444,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1212700,"kb_used_data":7056,"kb_used_omap":1,"kb_used_meta":1205630,"kb_avail":93159140,"statfs":{"total":96636764160,"available":95394959360,"internally_reserved":0,"allocated":7225344,"data_stored":3692309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1234565464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:49:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.52600000000000002,"15min":0.54100000000000004},"min":{"1min":0.317,"5min":0.20899999999999999,"15min":0.20100000000000001},"max":{"1min":0.74299999999999999,"5min":1.1060000000000001,"15min":3.2069999999999999},"last":0.626},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66300000000000003,"15min":0.747},"min":{"1min":0.40300000000000002,"5min":0.29199999999999998,"15min":0.222},"max":{"1min":0.81999999999999995,"5min":2.4209999999999998,"15min":10.430999999999999},"last":0.501}]},{"osd":2,"last update":"Mon Sep 16 13:49:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.23499999999999999,"15min":0.215},"max":{"1min":0.66500000000000004,"5min":1.6479999999999999,"15min":4.3090000000000002},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.70399999999999996,"15min":0.67900000000000005},"min":{"1min":0.42799999999999999,"5min":0.31,"15min":0.25},"max":{"1min":0.86499999999999999,"5min":1.6879999999999999,"15min":4.3869999999999996},"last":0.76700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:49:48.290 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-16T12:20:53.541416+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=20, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=49, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:49:48.290 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-16T12:20:53.541416+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=20, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=49, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:49:48.291 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-16T12:20:53.541416+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=20, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=49, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:49:48.291 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-16T12:20:53.541416+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=20, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=49, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:49:48.291 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-16T12:20:53.541416+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=20, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=49, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:49:48.291 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-16T12:20:53.541416+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=20, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=49, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:49:48.291 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-16T12:20:53.541416+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=20, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=49, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:49:48.291 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-16T12:20:53.541416+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=12, tm_min=20, tm_sec=53, tm_wday=0, tm_yday=260, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=16, tm_hour=13, tm_min=49, tm_sec=45, tm_wday=0, tm_yday=260, tm_isdst=0) 2024-09-16T13:49:48.291 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-16T13:50:08.292 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-16T13:50:08.598 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-09-16T13:50:08.601 INFO:teuthology.orchestra.run.smithi018.stderr:dumped all 2024-09-16T13:50:08.612 INFO:teuthology.orchestra.run.smithi018.stdout:{"pg_ready":true,"pg_map":{"version":4606,"stamp":"2024-09-16T13:50:07.972425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59,"ondisk_log_size":59,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3336064,"kb_used_data":20584,"kb_used_omap":4,"kb_used_meta":3315451,"kb_avail":279779456,"statfs":{"total":289910292480,"available":286494162944,"internally_reserved":0,"allocated":21078016,"data_stored":10486540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5112,"internal_metadata":3395021832},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003468"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":1937,"reported_epoch":1928,"state":"active+clean","last_fresh":"2024-09-16T13:49:53.574141+0000","last_change":"2024-09-16T13:49:53.574055+0000","last_active":"2024-09-16T13:49:53.574141+0000","last_peered":"2024-09-16T13:49:53.574141+0000","last_clean":"2024-09-16T13:49:53.574141+0000","last_became_active":"2024-09-16T12:20:54.569007+0000","last_became_peered":"2024-09-16T12:20:54.569007+0000","last_unstale":"2024-09-16T13:49:53.574141+0000","last_undegraded":"2024-09-16T13:49:53.574141+0000","last_fullsized":"2024-09-16T13:49:53.574141+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:49:53.573790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:49:53.573790+0000","last_clean_scrub_stamp":"2024-09-16T13:49:53.573790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:56:28.114768+0000","scrub_duration":0.0032597279999999999,"objects_trimmed":0,"snaptrim_duration":0.00055482299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1937,"reported_epoch":1928,"state":"active+clean","last_fresh":"2024-09-16T13:49:57.478431+0000","last_change":"2024-09-16T13:49:57.478355+0000","last_active":"2024-09-16T13:49:57.478431+0000","last_peered":"2024-09-16T13:49:57.478431+0000","last_clean":"2024-09-16T13:49:57.478431+0000","last_became_active":"2024-09-16T12:20:54.571536+0000","last_became_peered":"2024-09-16T12:20:54.571536+0000","last_unstale":"2024-09-16T13:49:57.478431+0000","last_undegraded":"2024-09-16T13:49:57.478431+0000","last_fullsized":"2024-09-16T13:49:57.478431+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:49:57.478094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:49:57.478094+0000","last_clean_scrub_stamp":"2024-09-16T13:49:57.478094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T23:11:15.681215+0000","scrub_duration":0.0039842560000000003,"objects_trimmed":0,"snaptrim_duration":0.00062020100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1937,"reported_epoch":1928,"state":"active+clean","last_fresh":"2024-09-16T13:50:00.494227+0000","last_change":"2024-09-16T13:50:00.494076+0000","last_active":"2024-09-16T13:50:00.494227+0000","last_peered":"2024-09-16T13:50:00.494227+0000","last_clean":"2024-09-16T13:50:00.494227+0000","last_became_active":"2024-09-16T12:20:54.566716+0000","last_became_peered":"2024-09-16T12:20:54.566716+0000","last_unstale":"2024-09-16T13:50:00.494227+0000","last_undegraded":"2024-09-16T13:50:00.494227+0000","last_fullsized":"2024-09-16T13:50:00.494227+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:50:00.493826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:50:00.493826+0000","last_clean_scrub_stamp":"2024-09-16T13:50:00.493826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T21:06:11.936695+0000","scrub_duration":0.0034481189999999999,"objects_trimmed":0,"snaptrim_duration":0.0015293030000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1937,"reported_epoch":1928,"state":"active+clean","last_fresh":"2024-09-16T13:49:51.574391+0000","last_change":"2024-09-16T13:49:51.574288+0000","last_active":"2024-09-16T13:49:51.574391+0000","last_peered":"2024-09-16T13:49:51.574391+0000","last_clean":"2024-09-16T13:49:51.574391+0000","last_became_active":"2024-09-16T12:20:54.570346+0000","last_became_peered":"2024-09-16T12:20:54.570346+0000","last_unstale":"2024-09-16T13:49:51.574391+0000","last_undegraded":"2024-09-16T13:49:51.574391+0000","last_fullsized":"2024-09-16T13:49:51.574391+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:49:51.573881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:49:51.573881+0000","last_clean_scrub_stamp":"2024-09-16T13:49:51.573881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:10:11.724661+0000","scrub_duration":0.0040202520000000002,"objects_trimmed":0,"snaptrim_duration":0.00086539699999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"1928'3","reported_seq":1941,"reported_epoch":1928,"state":"active+clean","last_fresh":"2024-09-16T13:49:49.967249+0000","last_change":"2024-09-16T13:49:49.967175+0000","last_active":"2024-09-16T13:49:49.967249+0000","last_peered":"2024-09-16T13:49:49.967249+0000","last_clean":"2024-09-16T13:49:49.967249+0000","last_became_active":"2024-09-16T12:20:54.567480+0000","last_became_peered":"2024-09-16T12:20:54.567480+0000","last_unstale":"2024-09-16T13:49:49.967249+0000","last_undegraded":"2024-09-16T13:49:49.967249+0000","last_fullsized":"2024-09-16T13:49:49.967249+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"1928'3","last_scrub_stamp":"2024-09-16T13:49:49.966922+0000","last_deep_scrub":"1928'3","last_deep_scrub_stamp":"2024-09-16T13:49:49.966922+0000","last_clean_scrub_stamp":"2024-09-16T13:49:49.966922+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:09:10.950437+0000","scrub_duration":0.0062809629999999997,"objects_trimmed":0,"snaptrim_duration":0.00089212600000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1938,"reported_epoch":1928,"state":"active+clean","last_fresh":"2024-09-16T13:49:50.255148+0000","last_change":"2024-09-16T13:49:50.255031+0000","last_active":"2024-09-16T13:49:50.255148+0000","last_peered":"2024-09-16T13:49:50.255148+0000","last_clean":"2024-09-16T13:49:50.255148+0000","last_became_active":"2024-09-16T12:20:54.564812+0000","last_became_peered":"2024-09-16T12:20:54.564812+0000","last_unstale":"2024-09-16T13:49:50.255148+0000","last_undegraded":"2024-09-16T13:49:50.255148+0000","last_fullsized":"2024-09-16T13:49:50.255148+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:49:50.254637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:49:50.254637+0000","last_clean_scrub_stamp":"2024-09-16T13:49:50.254637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T14:06:58.634656+0000","scrub_duration":0.0035017690000000001,"objects_trimmed":0,"snaptrim_duration":0.00093612199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1938,"reported_epoch":1928,"state":"active+clean","last_fresh":"2024-09-16T13:49:48.246171+0000","last_change":"2024-09-16T13:49:48.246072+0000","last_active":"2024-09-16T13:49:48.246171+0000","last_peered":"2024-09-16T13:49:48.246171+0000","last_clean":"2024-09-16T13:49:48.246171+0000","last_became_active":"2024-09-16T12:20:54.566954+0000","last_became_peered":"2024-09-16T12:20:54.566954+0000","last_unstale":"2024-09-16T13:49:48.246171+0000","last_undegraded":"2024-09-16T13:49:48.246171+0000","last_fullsized":"2024-09-16T13:49:48.246171+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-16T13:49:48.245692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-16T13:49:48.245692+0000","last_clean_scrub_stamp":"2024-09-16T13:49:48.245692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:29:23.287901+0000","scrub_duration":0.0029217959999999999,"objects_trimmed":0,"snaptrim_duration":0.00086513400000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"1928'2","reported_seq":1939,"reported_epoch":1928,"state":"active+clean","last_fresh":"2024-09-16T13:49:49.606290+0000","last_change":"2024-09-16T13:49:49.606204+0000","last_active":"2024-09-16T13:49:49.606290+0000","last_peered":"2024-09-16T13:49:49.606290+0000","last_clean":"2024-09-16T13:49:49.606290+0000","last_became_active":"2024-09-16T12:20:54.564938+0000","last_became_peered":"2024-09-16T12:20:54.564938+0000","last_unstale":"2024-09-16T13:49:49.606290+0000","last_undegraded":"2024-09-16T13:49:49.606290+0000","last_fullsized":"2024-09-16T13:49:49.606290+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"1928'2","last_scrub_stamp":"2024-09-16T13:49:49.605984+0000","last_deep_scrub":"1928'2","last_deep_scrub_stamp":"2024-09-16T13:49:49.605984+0000","last_clean_scrub_stamp":"2024-09-16T13:49:49.605984+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T20:28:24.641275+0000","scrub_duration":0.0069596679999999996,"objects_trimmed":0,"snaptrim_duration":0.0011678050000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"1928'54","reported_seq":2004,"reported_epoch":1928,"state":"active+clean","last_fresh":"2024-09-16T13:49:47.571633+0000","last_change":"2024-09-16T13:49:47.571514+0000","last_active":"2024-09-16T13:49:47.571633+0000","last_peered":"2024-09-16T13:49:47.571633+0000","last_clean":"2024-09-16T13:49:47.571633+0000","last_became_active":"2024-09-16T12:20:50.381536+0000","last_became_peered":"2024-09-16T12:20:50.381536+0000","last_unstale":"2024-09-16T13:49:47.571633+0000","last_undegraded":"2024-09-16T13:49:47.571633+0000","last_fullsized":"2024-09-16T13:49:47.571633+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"1928'54","last_scrub_stamp":"2024-09-16T13:49:47.571252+0000","last_deep_scrub":"1928'54","last_deep_scrub_stamp":"2024-09-16T13:49:47.571252+0000","last_clean_scrub_stamp":"2024-09-16T13:49:47.571252+0000","objects_scrubbed":2,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T17:43:56.513644+0000","scrub_duration":0.0061503119999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359739447,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":914512,"kb_used_data":6472,"kb_used_omap":1,"kb_used_meta":908030,"kb_avail":93457328,"statfs":{"total":96636764160,"available":95700303872,"internally_reserved":0,"allocated":6627328,"data_stored":3101922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":929823064},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.437,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.626,"5min":2.589,"15min":3.4620000000000002},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.56299999999999994,"5min":0.25,"15min":0.25},"max":{"1min":0.77700000000000002,"5min":2.698,"15min":3.5049999999999999},"last":0.60199999999999998}]},{"osd":1,"last update":"Mon Sep 16 13:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.48199999999999998,"5min":0.23400000000000001,"15min":0.20100000000000001},"max":{"1min":0.61699999999999999,"5min":3.194,"15min":3.194},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.73599999999999999,"15min":0.70299999999999996},"min":{"1min":0.60599999999999998,"5min":0.32800000000000001,"15min":0.23599999999999999},"max":{"1min":0.81599999999999995,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":0.74099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359739446,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1208856,"kb_used_data":7056,"kb_used_omap":1,"kb_used_meta":1201790,"kb_avail":93162984,"statfs":{"total":96636764160,"available":95398895616,"internally_reserved":0,"allocated":7225344,"data_stored":3692309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1230633304},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Sep 16 13:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.65800000000000003,"15min":0.59799999999999998},"min":{"1min":0.41099999999999998,"5min":0.32500000000000001,"15min":0.188},"max":{"1min":0.67700000000000005,"5min":4.8920000000000003,"15min":4.8920000000000003},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.621,"15min":0.63800000000000001},"min":{"1min":0.49399999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.69999999999999996,"5min":1.4299999999999999,"15min":2.222},"last":0.46600000000000003}]},{"osd":2,"last update":"Mon Sep 16 13:49:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.69699999999999995,"15min":0.67900000000000005},"min":{"1min":0.44800000000000001,"5min":0.23200000000000001,"15min":0.218},"max":{"1min":3.629,"5min":4.8479999999999999,"15min":4.8479999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.78300000000000003,"15min":0.71199999999999997},"min":{"1min":0.61399999999999999,"5min":0.27000000000000002,"15min":0.253},"max":{"1min":3.6669999999999998,"5min":4.9359999999999999,"15min":4.9359999999999999},"last":0.69299999999999995}]}]},{"osd":0,"up_from":8,"seq":34359739448,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1212696,"kb_used_data":7056,"kb_used_omap":1,"kb_used_meta":1205630,"kb_avail":93159144,"statfs":{"total":96636764160,"available":95394963456,"internally_reserved":0,"allocated":7225344,"data_stored":3692309,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1234565464},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Sep 16 13:49:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.52600000000000002,"15min":0.54100000000000004},"min":{"1min":0.317,"5min":0.20899999999999999,"15min":0.20100000000000001},"max":{"1min":0.74299999999999999,"5min":1.1060000000000001,"15min":3.2069999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.66300000000000003,"15min":0.747},"min":{"1min":0.40300000000000002,"5min":0.29199999999999998,"15min":0.222},"max":{"1min":0.81999999999999995,"5min":2.4209999999999998,"15min":10.430999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Mon Sep 16 13:49:39 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53800000000000003,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.23499999999999999,"15min":0.215},"max":{"1min":0.66500000000000004,"5min":1.6479999999999999,"15min":4.3090000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.70399999999999996,"15min":0.67900000000000005},"min":{"1min":0.42799999999999999,"5min":0.31,"15min":0.25},"max":{"1min":0.86499999999999999,"5min":1.6879999999999999,"15min":4.3869999999999996},"last":0.746}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-16T13:50:08.613 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-09-16T13:50:08.930 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-16T13:50:08.930 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-16T13:50:08.931 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-09-16T13:50:08.931 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:50:12.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 is failed for ~0s 2024-09-16T13:50:15.033 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-09-16T13:50:15.033 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-09-16T13:50:15.033 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:50:18.090 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 is failed for ~0s 2024-09-16T13:50:18.090 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 has been restored 2024-09-16T13:50:21.135 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-09-16T13:50:21.136 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-09-16T13:50:21.136 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:50:23.293 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 is failed for ~0s 2024-09-16T13:50:23.293 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.1 has been restored 2024-09-16T13:50:27.238 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-09-16T13:50:27.239 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-16T13:50:27.239 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-09-16T13:50:27.239 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:50:27.270 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-09-16T13:50:27.270 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-16T13:50:27.270 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-09-16T13:50:27.271 INFO:teuthology.orchestra.run:waiting for 300 2024-09-16T13:50:27.339 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-09-16T13:50:27.339 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-16T13:50:27.339 DEBUG:teuthology.orchestra.run.smithi018:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-16T13:50:27.401 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi018.front.sepia.ceph.com 2024-09-16T13:50:27.402 DEBUG:teuthology.orchestra.run.smithi018:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-09-16T13:50:27.695 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi018.front.sepia.ceph.com 2024-09-16T13:50:27.695 DEBUG:teuthology.orchestra.run.smithi018:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-09-16T13:50:28.033 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi018.front.sepia.ceph.com 2024-09-16T13:50:28.033 DEBUG:teuthology.orchestra.run.smithi018:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-09-16T13:50:28.258 INFO:tasks.ceph:Archiving mon data... 2024-09-16T13:50:28.258 DEBUG:teuthology.misc:Transferring archived files from smithi018:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907248/data/cluster1.mon.a.tgz 2024-09-16T13:50:28.258 DEBUG:teuthology.orchestra.run.smithi018:> mktemp 2024-09-16T13:50:28.281 INFO:teuthology.orchestra.run.smithi018.stdout:/tmp/tmp.GVS1SLeZov 2024-09-16T13:50:28.282 DEBUG:teuthology.orchestra.run.smithi018:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.GVS1SLeZov 2024-09-16T13:50:28.293 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 has been restored 2024-09-16T13:50:28.434 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0666 /tmp/tmp.GVS1SLeZov 2024-09-16T13:50:28.482 DEBUG:teuthology.orchestra.remote:smithi018:/tmp/tmp.GVS1SLeZov is 593KB 2024-09-16T13:50:28.550 DEBUG:teuthology.orchestra.run.smithi018:> rm -fr /tmp/tmp.GVS1SLeZov 2024-09-16T13:50:28.557 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-16T13:50:28.557 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T13:50:28.559 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-16T13:50:28.576 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-16T13:50:28.576 DEBUG:teuthology.orchestra.run.smithi018:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T13:50:28.617 DEBUG:teuthology.orchestra.run.smithi172:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-16T13:50:28.639 INFO:teuthology.orchestra.run.smithi018.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T13:50:28.687 INFO:teuthology.orchestra.run.smithi172.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-16T13:50:28.694 INFO:tasks.ceph:Archiving crash dumps... 2024-09-16T13:50:28.696 DEBUG:teuthology.misc:Transferring archived files from smithi018:/var/lib/ceph/crash to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907248/remote/smithi018/crash 2024-09-16T13:50:28.697 DEBUG:teuthology.orchestra.run.smithi018:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T13:50:28.715 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/crash to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907248/remote/smithi172/crash 2024-09-16T13:50:28.716 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-16T13:50:28.732 INFO:tasks.ceph:Compressing logs... 2024-09-16T13:50:28.732 DEBUG:teuthology.orchestra.run.smithi018:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T13:50:28.735 DEBUG:teuthology.orchestra.run.smithi172:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-16T13:50:28.754 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-09-16T13:50:28.754 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24701.log 2024-09-16T13:50:28.755 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27798.log 2024-09-16T13:50:28.756 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27879.log 2024-09-16T13:50:28.756 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.24701.log: /var/log/ceph/cluster1-client.admin.27798.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24701.log.gz 2024-09-16T13:50:28.756 INFO:teuthology.orchestra.run.smithi018.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27798.log.gz 2024-09-16T13:50:28.757 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21075.log 2024-09-16T13:50:28.757 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.27879.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-09-16T13:50:28.758 INFO:teuthology.orchestra.run.smithi018.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27879.log.gz 2024-09-16T13:50:28.758 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.21075.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24529.log 2024-09-16T13:50:28.758 INFO:teuthology.orchestra.run.smithi018.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21075.log.gz 2024-09-16T13:50:28.759 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27934.log 2024-09-16T13:50:28.759 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.24529.log: /var/log/ceph/cluster2-mgr.x.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24529.log.gz 2024-09-16T13:50:28.759 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-09-16T13:50:28.760 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24509.log 2024-09-16T13:50:28.760 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.27934.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27934.log.gz 2024-09-16T13:50:28.760 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24772.log 2024-09-16T13:50:28.764 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1.log: /var/log/ceph/cluster1-client.admin.24509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27906.log 2024-09-16T13:50:28.765 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24907.log 2024-09-16T13:50:28.765 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.24772.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24772.log.gz 2024-09-16T13:50:28.765 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.27906.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27906.log.gz 2024-09-16T13:50:28.765 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-09-16T13:50:28.766 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24395.log 2024-09-16T13:50:28.766 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.24907.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24907.log.gz 2024-09-16T13:50:28.767 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24618.log 2024-09-16T13:50:28.768 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.24395.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24395.log.gz 2024-09-16T13:50:28.772 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24615.log 2024-09-16T13:50:28.772 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.24618.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27717.log 2024-09-16T13:50:28.773 INFO:teuthology.orchestra.run.smithi018.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24618.log.gz 2024-09-16T13:50:28.773 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21127.log 2024-09-16T13:50:28.774 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.24615.log: /var/log/ceph/cluster1-client.admin.27717.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27717.log.gz 2024-09-16T13:50:28.774 INFO:teuthology.orchestra.run.smithi018.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24615.log.gz 2024-09-16T13:50:28.774 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27609.log 2024-09-16T13:50:28.774 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.21127.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21127.log.gz 2024-09-16T13:50:28.774 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27636.log 2024-09-16T13:50:28.775 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.27609.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27609.log.gz 2024-09-16T13:50:28.775 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21258.log 2024-09-16T13:50:28.775 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.27636.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27636.log.gz 2024-09-16T13:50:28.776 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21102.log 2024-09-16T13:50:28.776 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.21258.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21258.log.gz 2024-09-16T13:50:28.776 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-09-16T13:50:28.776 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.21102.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21102.log.gz 2024-09-16T13:50:28.776 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24695.log 2024-09-16T13:50:28.777 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21163.log 2024-09-16T13:50:28.777 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104620.log 2024-09-16T13:50:28.778 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24612.log 2024-09-16T13:50:28.778 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.24695.log: 0.0%/var/log/ceph/cluster1-client.admin.21163.log: -- replaced with /var/log/ceph/cluster1-client.admin.24695.log.gz 2024-09-16T13:50:28.778 INFO:teuthology.orchestra.run.smithi018.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21163.log.gz 2024-09-16T13:50:28.778 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-09-16T13:50:28.778 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.24612.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27825.log 2024-09-16T13:50:28.778 INFO:teuthology.orchestra.run.smithi018.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24612.log.gz 2024-09-16T13:50:28.782 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27959.log 2024-09-16T13:50:28.782 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-osd.2.log: /var/log/ceph/cluster1-client.admin.27825.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27825.log.gz 2024-09-16T13:50:28.782 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20980.log 2024-09-16T13:50:28.782 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27852.log 2024-09-16T13:50:28.782 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24780.log 2024-09-16T13:50:28.783 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.27959.log: /var/log/ceph/cluster1-client.admin.27852.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27959.log.gz 2024-09-16T13:50:28.783 INFO:teuthology.orchestra.run.smithi018.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27852.log.gz 2024-09-16T13:50:28.783 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.20980.log: 80.4% -- replaced with /var/log/ceph/cluster1-client.admin.24509.log.gz 2024-09-16T13:50:28.783 INFO:teuthology.orchestra.run.smithi018.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.20980.log.gz 2024-09-16T13:50:28.783 INFO:teuthology.orchestra.run.smithi018.stderr: 86.6% -- replaced with /var/log/ceph/cluster1.log.gz 2024-09-16T13:50:28.783 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27744.log 2024-09-16T13:50:28.784 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70334.log 2024-09-16T13:50:28.784 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77624.log 2024-09-16T13:50:28.785 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104620.log: /var/log/ceph/cluster1-client.mirror.70334.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133796.log 2024-09-16T13:50:28.785 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28707.log 2024-09-16T13:50:28.785 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77624.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141330.log 2024-09-16T13:50:28.785 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133796.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87308.log 2024-09-16T13:50:28.785 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28707.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134456.log 2024-09-16T13:50:28.785 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141330.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97289.log 2024-09-16T13:50:28.785 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87308.log: /var/log/ceph/cluster1-client.mirror.134456.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109341.log 2024-09-16T13:50:28.785 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112311.log 2024-09-16T13:50:28.785 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.97289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89509.log 2024-09-16T13:50:28.786 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.109341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63040.log 2024-09-16T13:50:28.786 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112311.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79141.log 2024-09-16T13:50:28.786 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105309.log 2024-09-16T13:50:28.791 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63040.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114074.log 2024-09-16T13:50:28.791 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79141.log: /var/log/ceph/cluster1-client.mirror.105309.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78192.log 2024-09-16T13:50:28.791 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40186.log 2024-09-16T13:50:28.791 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114074.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98772.log 2024-09-16T13:50:28.791 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78192.log: /var/log/ceph/cluster1-client.mirror.40186.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108589.log 2024-09-16T13:50:28.791 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104620.log.gz 2024-09-16T13:50:28.791 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98772.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66024.log 2024-09-16T13:50:28.792 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.108589.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28175.log 2024-09-16T13:50:28.792 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66024.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121993.log 2024-09-16T13:50:28.792 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96465.log 2024-09-16T13:50:28.792 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28175.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100201.log 2024-09-16T13:50:28.792 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.121993.log: /var/log/ceph/cluster2-client.admin.96465.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113823.log 2024-09-16T13:50:28.795 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.24780.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27663.log 2024-09-16T13:50:28.795 INFO:teuthology.orchestra.run.smithi018.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24780.log.gz 2024-09-16T13:50:28.795 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27690.log 2024-09-16T13:50:28.795 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.27663.log: /var/log/ceph/cluster1-client.admin.27744.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27663.log.gz 2024-09-16T13:50:28.795 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24483.log 2024-09-16T13:50:28.795 INFO:teuthology.orchestra.run.smithi018.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27744.log.gz 2024-09-16T13:50:28.795 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24692.log 2024-09-16T13:50:28.795 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.27690.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27690.log.gz 2024-09-16T13:50:28.795 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.24483.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.17355.log 2024-09-16T13:50:28.795 INFO:teuthology.orchestra.run.smithi018.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24483.log.gz 2024-09-16T13:50:28.796 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-09-16T13:50:28.796 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.24692.log: /var/log/ceph/ceph.tmp-client.admin.17355.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24692.log.gz 2024-09-16T13:50:28.796 INFO:teuthology.orchestra.run.smithi018.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.17355.log.gz 2024-09-16T13:50:28.796 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24457.log 2024-09-16T13:50:28.796 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24853.log 2024-09-16T13:50:28.796 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.24457.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27771.log 2024-09-16T13:50:28.796 INFO:teuthology.orchestra.run.smithi018.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24457.log.gz 2024-09-16T13:50:28.796 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.24853.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24853.log.gz 2024-09-16T13:50:28.796 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24556.log 2024-09-16T13:50:28.796 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.27771.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27771.log.gz 2024-09-16T13:50:28.796 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24583.log 2024-09-16T13:50:28.796 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.24556.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24556.log.gz 2024-09-16T13:50:28.797 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-09-16T13:50:28.797 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.24583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24776.log 2024-09-16T13:50:28.797 INFO:teuthology.orchestra.run.smithi018.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24583.log.gz 2024-09-16T13:50:28.797 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24880.log 2024-09-16T13:50:28.797 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.24776.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24776.log.gz 2024-09-16T13:50:28.797 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/cluster1-client.admin.24880.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.24880.log.gz 2024-09-16T13:50:28.797 INFO:teuthology.orchestra.run.smithi018.stderr: 93.1% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-09-16T13:50:28.802 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100201.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123369.log 2024-09-16T13:50:28.802 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113823.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76717.log 2024-09-16T13:50:28.802 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123369.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78930.log 2024-09-16T13:50:28.803 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose /var/log/ceph/cluster1-client.mirror.76717.log: -- /var/log/ceph/cluster1-client.mirror.67291.log 2024-09-16T13:50:28.803 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32569.log 2024-09-16T13:50:28.803 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78930.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92663.log 2024-09-16T13:50:28.803 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67291.log: /var/log/ceph/cluster1-client.mirror.32569.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99593.log 2024-09-16T13:50:28.803 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66446.log 2024-09-16T13:50:28.803 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92663.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96833.log 2024-09-16T13:50:28.803 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99593.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81264.log 2024-09-16T13:50:28.803 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66446.log: /var/log/ceph/cluster1-client.admin.96833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122291.log 2024-09-16T13:50:28.803 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.81264.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39555.log 2024-09-16T13:50:28.803 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26094.log 2024-09-16T13:50:28.803 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122291.log: /var/log/ceph/cluster1-client.mirror.39555.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42393.log 2024-09-16T13:50:28.803 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31819.log 2024-09-16T13:50:28.803 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.26094.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69816.log 2024-09-16T13:50:28.803 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.26094.log.gz 2024-09-16T13:50:28.804 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42393.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77624.log.gz 2024-09-16T13:50:28.804 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55141.log 2024-09-16T13:50:28.804 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31819.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.97289.log.gz 2024-09-16T13:50:28.804 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90433.log 2024-09-16T13:50:28.804 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69816.log: /var/log/ceph/cluster1-client.mirror.55141.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85071.log 2024-09-16T13:50:28.804 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90433.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31690.log 2024-09-16T13:50:28.804 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85071.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40640.log 2024-09-16T13:50:28.804 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31690.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.87308.log.gz 2024-09-16T13:50:28.804 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106711.log 2024-09-16T13:50:28.809 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40640.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99593.log.gz 2024-09-16T13:50:28.809 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114453.log 2024-09-16T13:50:28.809 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106711.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58125.log 2024-09-16T13:50:28.809 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139668.log 2024-09-16T13:50:28.809 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58125.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108341.log 2024-09-16T13:50:28.810 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139668.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129398.log 2024-09-16T13:50:28.810 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.108341.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78192.log.gz 2024-09-16T13:50:28.810 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64110.log 2024-09-16T13:50:28.810 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78511.log 2024-09-16T13:50:28.810 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129398.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89509.log.gz 2024-09-16T13:50:28.810 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64110.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118951.log 2024-09-16T13:50:28.810 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127846.log 2024-09-16T13:50:28.810 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78511.log: /var/log/ceph/cluster2-client.admin.118951.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54236.log 2024-09-16T13:50:28.810 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127846.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63894.log 2024-09-16T13:50:28.810 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79141.log.gz 2024-09-16T13:50:28.812 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54236.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116286.log 2024-09-16T13:50:28.813 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61064.log 2024-09-16T13:50:28.813 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63894.log: /var/log/ceph/cluster1-client.mirror.116286.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81264.log.gz 2024-09-16T13:50:28.813 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50688.log 2024-09-16T13:50:28.813 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61064.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122880.log 2024-09-16T13:50:28.813 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50688.log: /var/log/ceph/cluster1-client.mirror.52963.log 2024-09-16T13:50:28.813 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69812.log 2024-09-16T13:50:28.813 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109464.log 2024-09-16T13:50:28.814 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69812.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73060.log 2024-09-16T13:50:28.814 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124186.log 2024-09-16T13:50:28.815 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109464.log: /var/log/ceph/cluster2-client.admin.73060.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55562.log 2024-09-16T13:50:28.815 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73060.log.gz 2024-09-16T13:50:28.815 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78930.log.gz 2024-09-16T13:50:28.815 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51772.log 2024-09-16T13:50:28.815 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124186.log: /var/log/ceph/cluster1-client.mirror.55562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86752.log 2024-09-16T13:50:28.816 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119071.log 2024-09-16T13:50:28.817 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.51772.log: /var/log/ceph/cluster1-client.mirror.86752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29549.log 2024-09-16T13:50:28.817 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.119071.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69397.log 2024-09-16T13:50:28.817 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29549.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117054.log 2024-09-16T13:50:28.818 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55141.log.gz 2024-09-16T13:50:28.818 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69397.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97000.log 2024-09-16T13:50:28.818 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72676.log 2024-09-16T13:50:28.819 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.117054.log: /var/log/ceph/cluster2-client.mirror.97000.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114390.log 2024-09-16T13:50:28.819 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66446.log.gz 2024-09-16T13:50:28.819 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71273.log 2024-09-16T13:50:28.820 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.72676.log: /var/log/ceph/cluster1-client.mirror.114390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143334.log 2024-09-16T13:50:28.820 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71273.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51013.log 2024-09-16T13:50:28.821 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.143334.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40186.log.gz 2024-09-16T13:50:28.821 INFO:teuthology.orchestra.run.smithi172.stderr:gzip 87.1%-5 --verbose -- replaced with /var/log/ceph/cluster1-client.admin.143334.log.gz-- 2024-09-16T13:50:28.821 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.60310.log 2024-09-16T13:50:28.822 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103757.log 2024-09-16T13:50:28.822 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51013.log: /var/log/ceph/cluster2-client.admin.60310.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87845.log 2024-09-16T13:50:28.823 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103757.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60313.log 2024-09-16T13:50:28.824 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32569.log.gz 2024-09-16T13:50:28.824 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87845.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43233.log 2024-09-16T13:50:28.824 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69397.log.gz 2024-09-16T13:50:28.825 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.60313.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131094.log 2024-09-16T13:50:28.825 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43233.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92199.log 2024-09-16T13:50:28.826 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.131094.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127822.log 2024-09-16T13:50:28.826 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131094.log.gz 2024-09-16T13:50:28.827 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39555.log.gz 2024-09-16T13:50:28.827 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92199.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121123.log 2024-09-16T13:50:28.828 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127822.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92832.log 2024-09-16T13:50:28.828 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121123.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60200.log 2024-09-16T13:50:28.828 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42393.log.gz 2024-09-16T13:50:28.829 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92832.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54928.log 2024-09-16T13:50:28.829 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.60200.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87720.log 2024-09-16T13:50:28.830 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54928.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61493.log 2024-09-16T13:50:28.830 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.87720.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66024.log.gz 2024-09-16T13:50:28.830 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77016.log 2024-09-16T13:50:28.830 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61493.log: gzip 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.61493.log.gz 2024-09-16T13:50:28.830 INFO:teuthology.orchestra.run.smithi172.stderr:-5 --verbose -- /var/log/ceph/cluster2-client.admin.141791.log 2024-09-16T13:50:28.831 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77016.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28888.log 2024-09-16T13:50:28.832 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139993.log 2024-09-16T13:50:28.832 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.141791.log: /var/log/ceph/cluster1-client.mirror.28888.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25669.log 2024-09-16T13:50:28.832 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58125.log.gz 2024-09-16T13:50:28.832 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.139993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126559.log 2024-09-16T13:50:28.834 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.139993.log.gz 2024-09-16T13:50:28.834 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93210.log 2024-09-16T13:50:28.834 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63040.log.gz 2024-09-16T13:50:28.834 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.25669.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86773.log 2024-09-16T13:50:28.834 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126559.log: /var/log/ceph/cluster1-client.mirror.93210.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37516.log 2024-09-16T13:50:28.835 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.86773.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86773.log.gz 2024-09-16T13:50:28.835 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136492.log 2024-09-16T13:50:28.835 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37516.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37516.log.gz 2024-09-16T13:50:28.836 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136850.log 2024-09-16T13:50:28.836 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136492.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84756.log 2024-09-16T13:50:28.837 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136850.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.28707.log.gz 2024-09-16T13:50:28.837 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31689.log 2024-09-16T13:50:28.838 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.84756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128994.log 2024-09-16T13:50:28.839 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121876.log 2024-09-16T13:50:28.839 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31689.log: /var/log/ceph/cluster1-client.admin.128994.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40420.log 2024-09-16T13:50:28.839 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121876.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92663.log.gz 2024-09-16T13:50:28.840 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28084.log 2024-09-16T13:50:28.841 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40420.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91775.log 2024-09-16T13:50:28.843 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44641.log 2024-09-16T13:50:28.843 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52569.log 2024-09-16T13:50:28.843 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91775.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105739.log 2024-09-16T13:50:28.844 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44641.log: /var/log/ceph/cluster1-client.mirror.52569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61055.log 2024-09-16T13:50:28.844 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78511.log.gz 2024-09-16T13:50:28.844 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43232.log 2024-09-16T13:50:28.844 INFO:teuthology.orchestra.run.smithi172.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108341.log.gz 2024-09-16T13:50:28.844 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61055.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.61055.log.gz 2024-09-16T13:50:28.846 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112310.log 2024-09-16T13:50:28.846 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43232.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108033.log 2024-09-16T13:50:28.846 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.85071.log.gz 2024-09-16T13:50:28.846 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91946.log 2024-09-16T13:50:28.846 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112310.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134387.log 2024-09-16T13:50:28.846 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108033.log: /var/log/ceph/cluster1-client.mirror.91946.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141250.log 2024-09-16T13:50:28.847 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134387.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114518.log 2024-09-16T13:50:28.848 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141250.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60689.log 2024-09-16T13:50:28.849 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63502.log 2024-09-16T13:50:28.849 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114518.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41191.log 2024-09-16T13:50:28.849 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.60689.log: /var/log/ceph/cluster1-client.mirror.63502.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72775.log 2024-09-16T13:50:28.850 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36292.log 2024-09-16T13:50:28.850 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41191.log: /var/log/ceph/cluster2-client.admin.72775.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80826.log 2024-09-16T13:50:28.851 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36292.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44117.log 2024-09-16T13:50:28.851 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114074.log.gz 2024-09-16T13:50:28.852 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121771.log 2024-09-16T13:50:28.852 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80826.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89485.log 2024-09-16T13:50:28.853 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44117.log: /var/log/ceph/cluster1-client.admin.121771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92705.log 2024-09-16T13:50:28.853 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75351.log 2024-09-16T13:50:28.853 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89485.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39619.log 2024-09-16T13:50:28.854 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92705.log: /var/log/ceph/cluster1-client.admin.75351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123909.log 2024-09-16T13:50:28.854 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75351.log.gz 2024-09-16T13:50:28.854 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39619.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49664.log 2024-09-16T13:50:28.854 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123909.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51490.log 2024-09-16T13:50:28.856 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.49664.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139165.log 2024-09-16T13:50:28.856 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51490.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69393.log 2024-09-16T13:50:28.856 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70349.log 2024-09-16T13:50:28.857 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69393.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118721.log 2024-09-16T13:50:28.858 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.70349.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68951.log 2024-09-16T13:50:28.858 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118721.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130049.log 2024-09-16T13:50:28.861 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68951.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67078.log 2024-09-16T13:50:28.862 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130049.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104397.log 2024-09-16T13:50:28.862 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133796.log.gz 2024-09-16T13:50:28.862 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67078.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61965.log 2024-09-16T13:50:28.862 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113865.log 2024-09-16T13:50:28.862 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104397.log: /var/log/ceph/cluster1-client.admin.61965.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33411.log 2024-09-16T13:50:28.862 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128340.log 2024-09-16T13:50:28.868 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113865.log: /var/log/ceph/cluster1-client.mirror.33411.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57451.log 2024-09-16T13:50:28.868 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.112311.log.gz 2024-09-16T13:50:28.868 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.128340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84544.log 2024-09-16T13:50:28.868 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112310.log.gz 2024-09-16T13:50:28.869 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57451.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73174.log 2024-09-16T13:50:28.869 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110714.log 2024-09-16T13:50:28.869 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.84544.log: /var/log/ceph/cluster2-client.admin.73174.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63565.log 2024-09-16T13:50:28.869 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113823.log.gz 2024-09-16T13:50:28.869 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26782.log 2024-09-16T13:50:28.869 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110714.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111722.log 2024-09-16T13:50:28.869 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63565.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134456.log.gz 2024-09-16T13:50:28.869 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.26782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109604.log 2024-09-16T13:50:28.869 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56692.log 2024-09-16T13:50:28.869 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111722.log: /var/log/ceph/cluster1-client.mirror.109604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126920.log 2024-09-16T13:50:28.869 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50662.log 2024-09-16T13:50:28.869 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122291.log.gz 2024-09-16T13:50:28.870 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126920.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126285.log 2024-09-16T13:50:28.871 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.50662.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78276.log 2024-09-16T13:50:28.871 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32872.log 2024-09-16T13:50:28.871 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126285.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63154.log 2024-09-16T13:50:28.871 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78276.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91481.log 2024-09-16T13:50:28.872 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32872.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40251.log 2024-09-16T13:50:28.872 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.63154.log: /var/log/ceph/cluster1-client.mirror.91481.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105309.log.gz 2024-09-16T13:50:28.872 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72164.log 2024-09-16T13:50:28.873 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40251.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92664.log 2024-09-16T13:50:28.874 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40743.log 2024-09-16T13:50:28.874 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.72164.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81012.log 2024-09-16T13:50:28.874 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92664.log: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98772.log.gz 2024-09-16T13:50:28.874 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67706.log 2024-09-16T13:50:28.874 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33582.log 2024-09-16T13:50:28.874 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101232.log 2024-09-16T13:50:28.875 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67706.log: /var/log/ceph/cluster1-client.mirror.33582.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72648.log 2024-09-16T13:50:28.875 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49122.log 2024-09-16T13:50:28.875 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101232.log: /var/log/ceph/cluster2-client.admin.72648.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115151.log 2024-09-16T13:50:28.875 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49122.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49122.log.gz 2024-09-16T13:50:28.876 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48707.log 2024-09-16T13:50:28.876 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.96833.log.gz 2024-09-16T13:50:28.876 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115151.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113972.log 2024-09-16T13:50:28.877 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.48707.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78152.log 2024-09-16T13:50:28.877 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113972.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43692.log 2024-09-16T13:50:28.877 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78152.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74362.log 2024-09-16T13:50:28.878 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87272.log 2024-09-16T13:50:28.878 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43692.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91775.log.gz 2024-09-16T13:50:28.878 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.74362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54865.log 2024-09-16T13:50:28.879 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143314.log 2024-09-16T13:50:28.880 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54865.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126077.log 2024-09-16T13:50:28.882 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87845.log.gz 2024-09-16T13:50:28.882 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77101.log 2024-09-16T13:50:28.882 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.143314.log: /var/log/ceph/cluster1-client.mirror.126077.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87243.log 2024-09-16T13:50:28.882 INFO:teuthology.orchestra.run.smithi172.stderr: 68.3% -- replaced with /var/log/ceph/cluster1-client.mirror.143314.log.gz 2024-09-16T13:50:28.882 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89174.log 2024-09-16T13:50:28.883 INFO:teuthology.orchestra.run.smithi172.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.109341.log.gz 2024-09-16T13:50:28.883 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87243.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45588.log 2024-09-16T13:50:28.883 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109464.log.gz 2024-09-16T13:50:28.883 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89174.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63818.log 2024-09-16T13:50:28.884 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45588.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66444.log 2024-09-16T13:50:28.884 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63818.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93126.log 2024-09-16T13:50:28.884 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72328.log 2024-09-16T13:50:28.884 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66444.log: /var/log/ceph/cluster1-client.mirror.93126.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67709.log 2024-09-16T13:50:28.885 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69734.log 2024-09-16T13:50:28.885 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.72328.log: /var/log/ceph/cluster1-client.mirror.67709.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41972.log 2024-09-16T13:50:28.885 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72044.log 2024-09-16T13:50:28.886 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69734.log: /var/log/ceph/cluster1-client.mirror.41972.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62759.log 2024-09-16T13:50:28.886 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105739.log.gz 2024-09-16T13:50:28.888 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.72044.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134537.log 2024-09-16T13:50:28.888 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.62759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28770.log 2024-09-16T13:50:28.888 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63347.log 2024-09-16T13:50:28.889 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126372.log 2024-09-16T13:50:28.889 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.28770.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28770.log.gz 2024-09-16T13:50:28.889 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63347.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39796.log 2024-09-16T13:50:28.892 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.126372.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46283.log 2024-09-16T13:50:28.892 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose /var/log/ceph/cluster1-client.admin.39796.log: -- /var/log/ceph/cluster1-client.mirror.41912.log 2024-09-16T13:50:28.892 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139435.log 2024-09-16T13:50:28.892 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40640.log.gz 2024-09-16T13:50:28.892 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46283.log: /var/log/ceph/cluster1-client.mirror.41912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136072.log 2024-09-16T13:50:28.892 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139435.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79241.log 2024-09-16T13:50:28.893 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136072.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113106.log 2024-09-16T13:50:28.894 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79241.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42812.log 2024-09-16T13:50:28.894 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31899.log 2024-09-16T13:50:28.894 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42812.log: /var/log/ceph/cluster1-client.mirror.113106.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122750.log 2024-09-16T13:50:28.894 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124186.log.gz 2024-09-16T13:50:28.895 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31899.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67080.log 2024-09-16T13:50:28.896 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.118951.log.gz 2024-09-16T13:50:28.896 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113977.log 2024-09-16T13:50:28.896 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67080.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68845.log 2024-09-16T13:50:28.896 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122750.log: /var/log/ceph/cluster1-client.mirror.113977.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127212.log 2024-09-16T13:50:28.897 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68845.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129939.log 2024-09-16T13:50:28.897 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127212.log: 19.0% -- replaced with /var/log/ceph/cluster1-client.admin.121771.log.gz 2024-09-16T13:50:28.897 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40449.log 2024-09-16T13:50:28.897 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.129939.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84375.log 2024-09-16T13:50:28.898 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.40449.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49708.log 2024-09-16T13:50:28.899 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.84375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33713.log 2024-09-16T13:50:28.899 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49708.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81560.log 2024-09-16T13:50:28.900 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33713.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128340.log.gz 2024-09-16T13:50:28.900 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100117.log 2024-09-16T13:50:28.900 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84949.log 2024-09-16T13:50:28.901 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81560.log: /var/log/ceph/cluster1-client.mirror.100117.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127822.log.gz 2024-09-16T13:50:28.902 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141988.log 2024-09-16T13:50:28.902 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127846.log.gz 2024-09-16T13:50:28.902 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93167.log 2024-09-16T13:50:28.902 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.84949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76568.log 2024-09-16T13:50:28.903 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.141988.log: /var/log/ceph/cluster1-client.mirror.93167.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43443.log 2024-09-16T13:50:28.904 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53815.log 2024-09-16T13:50:28.904 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76568.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44221.log 2024-09-16T13:50:28.904 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43443.log: /var/log/ceph/cluster1-client.mirror.53815.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53816.log 2024-09-16T13:50:28.905 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84381.log 2024-09-16T13:50:28.905 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44221.log: /var/log/ceph/cluster1-client.mirror.53816.log: gzip -5 --verbose -- 89.9%/var/log/ceph/cluster1-client.mirror.139583.log 2024-09-16T13:50:28.905 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.116286.log.gz 2024-09-16T13:50:28.906 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60312.log 2024-09-16T13:50:28.906 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.84381.log: gzip /var/log/ceph/cluster1-client.mirror.139583.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.47892.log 2024-09-16T13:50:28.909 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.129398.log.gz 2024-09-16T13:50:28.909 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.60312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69524.log 2024-09-16T13:50:28.909 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.47892.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31190.log 2024-09-16T13:50:28.909 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30770.log 2024-09-16T13:50:28.909 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69524.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84024.log 2024-09-16T13:50:28.909 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31190.log: /var/log/ceph/cluster1-client.mirror.30770.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132426.log 2024-09-16T13:50:28.909 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.84024.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67289.log 2024-09-16T13:50:28.911 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126559.log.gz 2024-09-16T13:50:28.912 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49504.log 2024-09-16T13:50:28.912 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65605.log 2024-09-16T13:50:28.912 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.141791.log.gz 2024-09-16T13:50:28.912 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.49504.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85532.log 2024-09-16T13:50:28.912 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27628.log 2024-09-16T13:50:28.912 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65605.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38662.log 2024-09-16T13:50:28.912 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85532.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85532.log.gz 2024-09-16T13:50:28.912 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.27628.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131252.log 2024-09-16T13:50:28.913 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.38662.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27023.log 2024-09-16T13:50:28.913 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136492.log.gz 2024-09-16T13:50:28.913 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.131252.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53395.log 2024-09-16T13:50:28.914 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.27023.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114560.log 2024-09-16T13:50:28.914 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53395.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90267.log 2024-09-16T13:50:28.915 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96271.log 2024-09-16T13:50:28.915 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114560.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101884.log 2024-09-16T13:50:28.916 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90267.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85563.log 2024-09-16T13:50:28.916 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96271.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123369.log.gzgzip 2024-09-16T13:50:28.916 INFO:teuthology.orchestra.run.smithi172.stderr:-5 --verbose -- /var/log/ceph/cluster2-client.mirror.37352.log 2024-09-16T13:50:28.917 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101884.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112208.log 2024-09-16T13:50:28.917 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85563.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.141250.log.gz 2024-09-16T13:50:28.917 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85563.log.gz 2024-09-16T13:50:28.917 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.37352.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50863.log 2024-09-16T13:50:28.918 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34214.log 2024-09-16T13:50:28.918 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112208.log: gzip /var/log/ceph/cluster1-client.admin.50863.log: -5 --verbose -- /var/log/ceph/cluster2-client.admin.61117.log 2024-09-16T13:50:28.918 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34214.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108885.log 2024-09-16T13:50:28.920 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61117.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117054.log.gz 2024-09-16T13:50:28.920 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67506.log 2024-09-16T13:50:28.920 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.108885.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91566.log 2024-09-16T13:50:28.921 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67506.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139668.log.gz 2024-09-16T13:50:28.922 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87509.log 2024-09-16T13:50:28.922 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91566.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99457.log 2024-09-16T13:50:28.923 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87509.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85406.log 2024-09-16T13:50:28.923 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99457.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.97000.log.gz 2024-09-16T13:50:28.923 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105230.log 2024-09-16T13:50:28.924 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85406.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85406.log.gz 2024-09-16T13:50:28.924 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69895.log 2024-09-16T13:50:28.925 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105230.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81140.log 2024-09-16T13:50:28.925 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79242.log 2024-09-16T13:50:28.931 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113258.log 2024-09-16T13:50:28.931 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79242.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100492.log 2024-09-16T13:50:28.931 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44220.log 2024-09-16T13:50:28.931 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67080.log.gz 2024-09-16T13:50:28.931 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100492.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36833.log 2024-09-16T13:50:28.932 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0%/var/log/ceph/cluster1-client.mirror.44220.log: -- replaced with /var/log/ceph/cluster2-client.admin.61064.log.gz 2024-09-16T13:50:28.932 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111666.log 2024-09-16T13:50:28.932 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.36833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64369.log 2024-09-16T13:50:28.932 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111666.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89090.log 2024-09-16T13:50:28.932 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96133.log 2024-09-16T13:50:28.932 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64369.log: /var/log/ceph/cluster1-client.mirror.89090.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34682.log 2024-09-16T13:50:28.932 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.96133.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59554.log 2024-09-16T13:50:28.932 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34682.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110227.log 2024-09-16T13:50:28.933 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55562.log.gz 2024-09-16T13:50:28.933 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59554.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85084.log 2024-09-16T13:50:28.934 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110227.log: /var/log/ceph/cluster1-client.admin.83617.log 2024-09-16T13:50:28.934 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86146.log 2024-09-16T13:50:28.934 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.83617.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117157.log 2024-09-16T13:50:28.935 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.86146.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37548.log 2024-09-16T13:50:28.935 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66447.log 2024-09-16T13:50:28.936 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.117157.log: 18.0% -- replaced with /var/log/ceph/cluster2-client.admin.72044.log.gz 2024-09-16T13:50:28.936 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110828.log 2024-09-16T13:50:28.938 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96018.log 2024-09-16T13:50:28.938 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66447.log: /var/log/ceph/cluster1-client.admin.110828.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68971.log 2024-09-16T13:50:28.938 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51013.log.gz 2024-09-16T13:50:28.938 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.96018.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104955.log 2024-09-16T13:50:28.939 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72348.log 2024-09-16T13:50:28.940 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68971.log: /var/log/ceph/cluster1-client.mirror.104955.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123512.log 2024-09-16T13:50:28.940 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86655.log 2024-09-16T13:50:28.940 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.60200.log.gz 2024-09-16T13:50:28.940 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.72348.log: /var/log/ceph/cluster1-client.mirror.123512.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124749.log 2024-09-16T13:50:28.942 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.86655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40704.log 2024-09-16T13:50:28.942 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96947.log 2024-09-16T13:50:28.942 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124749.log: /var/log/ceph/cluster1-client.admin.40704.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121904.log 2024-09-16T13:50:28.942 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.96947.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130467.log 2024-09-16T13:50:28.942 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33622.log 2024-09-16T13:50:28.943 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130467.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38205.log 2024-09-16T13:50:28.944 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33622.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117017.log 2024-09-16T13:50:28.944 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.38205.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49706.log 2024-09-16T13:50:28.944 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.51772.log.gz 2024-09-16T13:50:28.944 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117017.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28212.log 2024-09-16T13:50:28.944 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.117017.log.gz 2024-09-16T13:50:28.944 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109487.log 2024-09-16T13:50:28.945 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49706.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49706.log.gz 2024-09-16T13:50:28.945 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28212.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71898.log 2024-09-16T13:50:28.945 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.109487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104596.log 2024-09-16T13:50:28.947 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63677.log 2024-09-16T13:50:28.947 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.71898.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85547.log 2024-09-16T13:50:28.948 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52963.log.gz 2024-09-16T13:50:28.948 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104596.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95781.log 2024-09-16T13:50:28.948 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63677.log: /var/log/ceph/cluster2-client.admin.85547.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122753.log 2024-09-16T13:50:28.948 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85547.log.gz 2024-09-16T13:50:28.948 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58167.log 2024-09-16T13:50:28.948 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.95781.log: /var/log/ceph/cluster1-client.mirror.122753.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74289.log 2024-09-16T13:50:28.949 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58167.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49231.log 2024-09-16T13:50:28.949 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.74289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26799.log 2024-09-16T13:50:28.949 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31819.log.gz 2024-09-16T13:50:28.949 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49231.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39881.log 2024-09-16T13:50:28.950 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.26799.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.36774.log 2024-09-16T13:50:28.950 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142529.log 2024-09-16T13:50:28.951 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44955.log 2024-09-16T13:50:28.952 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36774.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36677.log 2024-09-16T13:50:28.952 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142529.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142529.log.gz 2024-09-16T13:50:28.952 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44955.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75802.log 2024-09-16T13:50:28.952 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.86752.log.gz 2024-09-16T13:50:28.952 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36677.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94940.log 2024-09-16T13:50:28.953 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95217.log 2024-09-16T13:50:28.954 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75802.log: /var/log/ceph/cluster1-client.admin.94940.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50594.log 2024-09-16T13:50:28.954 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63607.log 2024-09-16T13:50:28.954 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.95217.log: /var/log/ceph/cluster2-client.mirror.50594.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102914.log 2024-09-16T13:50:28.955 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.63607.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77856.log 2024-09-16T13:50:28.955 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78152.log.gz 2024-09-16T13:50:28.955 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102914.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125294.log 2024-09-16T13:50:28.955 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77856.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78276.log.gz 2024-09-16T13:50:28.956 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42076.log 2024-09-16T13:50:28.956 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39642.log 2024-09-16T13:50:28.957 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42076.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.84375.log.gz 2024-09-16T13:50:28.957 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27374.log 2024-09-16T13:50:28.958 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41294.log 2024-09-16T13:50:28.958 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.39642.log: /var/log/ceph/cluster1-client.admin.27374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27968.log 2024-09-16T13:50:28.958 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27374.log.gz 2024-09-16T13:50:28.959 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112941.log 2024-09-16T13:50:28.959 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52115.log 2024-09-16T13:50:28.960 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112941.log: /var/log/ceph/cluster1-client.admin.27968.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27968.log.gz 2024-09-16T13:50:28.960 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142674.log 2024-09-16T13:50:28.961 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.52115.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126794.log 2024-09-16T13:50:28.961 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142674.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34954.log 2024-09-16T13:50:28.961 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126794.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90263.log 2024-09-16T13:50:28.962 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.34954.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34954.log.gz 2024-09-16T13:50:28.962 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100201.log.gz 2024-09-16T13:50:28.963 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114310.log 2024-09-16T13:50:28.963 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90263.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87243.log.gz 2024-09-16T13:50:28.964 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114181.log 2024-09-16T13:50:28.965 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132077.log 2024-09-16T13:50:28.965 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114310.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77016.log.gz 2024-09-16T13:50:28.965 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114181.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118744.log 2024-09-16T13:50:28.966 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85690.log 2024-09-16T13:50:28.966 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.132077.log: gzip -5 --verbose /var/log/ceph/cluster2-client.admin.118744.log: -- /var/log/ceph/cluster1-client.mirror.54864.log 2024-09-16T13:50:28.966 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77436.log 2024-09-16T13:50:28.967 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.85690.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97134.log 2024-09-16T13:50:28.967 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104955.log.gz 2024-09-16T13:50:28.967 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54864.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85238.log /var/log/ceph/cluster1-client.mirror.77436.log: 2024-09-16T13:50:28.968 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.97134.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132954.log 2024-09-16T13:50:28.969 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137059.log 2024-09-16T13:50:28.969 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85238.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80930.log 2024-09-16T13:50:28.969 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.132954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140149.log 2024-09-16T13:50:28.970 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137059.log: gzip /var/log/ceph/cluster1-client.mirror.80930.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111934.log 2024-09-16T13:50:28.970 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90433.log.gz 2024-09-16T13:50:28.970 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.140149.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105719.log 2024-09-16T13:50:28.971 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28593.log 2024-09-16T13:50:28.971 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111934.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78402.log 2024-09-16T13:50:28.971 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.105719.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105719.log.gz 2024-09-16T13:50:28.972 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.83617.log.gz 2024-09-16T13:50:28.972 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.28593.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28593.log.gz 2024-09-16T13:50:28.972 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46072.log 2024-09-16T13:50:28.972 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78402.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103016.log 2024-09-16T13:50:28.973 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46072.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81560.log.gz 2024-09-16T13:50:28.974 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139973.log 2024-09-16T13:50:28.974 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103016.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69814.log 2024-09-16T13:50:28.975 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116558.log 2024-09-16T13:50:28.975 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.139973.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.139973.log.gz 2024-09-16T13:50:28.975 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32449.log 2024-09-16T13:50:28.975 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69814.log: /var/log/ceph/cluster1-client.mirror.116558.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129307.log 2024-09-16T13:50:28.976 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32449.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93504.log 2024-09-16T13:50:28.976 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44744.log 2024-09-16T13:50:28.976 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38703.log 2024-09-16T13:50:28.977 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93504.log: /var/log/ceph/cluster1-client.mirror.44744.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100787.log 2024-09-16T13:50:28.978 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.99457.log.gz 2024-09-16T13:50:28.978 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.38703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53880.log 2024-09-16T13:50:28.978 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100787.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75020.log 2024-09-16T13:50:28.979 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104072.log 2024-09-16T13:50:28.979 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.96133.log.gz 2024-09-16T13:50:28.980 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.75020.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42962.log 2024-09-16T13:50:28.980 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75020.log.gz 2024-09-16T13:50:28.981 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93797.log 2024-09-16T13:50:28.981 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104072.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25859.log 2024-09-16T13:50:28.981 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42962.log: /var/log/ceph/cluster1-client.mirror.93797.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106970.log 2024-09-16T13:50:28.982 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25859.log: /var/log/ceph/cluster1-client.admin.83837.log 2024-09-16T13:50:28.983 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25859.log.gz 2024-09-16T13:50:28.983 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106970.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80613.log 2024-09-16T13:50:28.983 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.84756.log.gz/var/log/ceph/cluster1-client.admin.83837.log: 2024-09-16T13:50:28.983 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48556.log 2024-09-16T13:50:28.983 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39689.log 2024-09-16T13:50:28.984 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80613.log: /var/log/ceph/cluster2-client.admin.48556.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42012.log 2024-09-16T13:50:28.985 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101884.log.gz 2024-09-16T13:50:28.985 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39689.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78193.log 2024-09-16T13:50:28.985 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90896.log 2024-09-16T13:50:28.985 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41490.log 2024-09-16T13:50:28.986 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78193.log: gzip /var/log/ceph/cluster1-client.mirror.90896.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122538.log 2024-09-16T13:50:28.986 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71456.log 2024-09-16T13:50:28.986 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41490.log: /var/log/ceph/cluster1-client.mirror.122538.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124831.log 2024-09-16T13:50:28.986 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76717.log.gz 2024-09-16T13:50:28.986 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71456.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62474.log 2024-09-16T13:50:28.987 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124831.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118487.log 2024-09-16T13:50:28.987 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68974.log 2024-09-16T13:50:28.987 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.62474.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97971.log 2024-09-16T13:50:28.988 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.118487.log: /var/log/ceph/cluster1-client.mirror.68974.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144379.log 2024-09-16T13:50:28.989 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.97971.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97725.log 2024-09-16T13:50:28.990 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.144379.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144379.log.gz 2024-09-16T13:50:28.990 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115822.log 2024-09-16T13:50:28.990 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113865.log.gz 2024-09-16T13:50:28.992 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.97725.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108610.log 2024-09-16T13:50:28.992 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134927.log 2024-09-16T13:50:28.992 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115822.log: /var/log/ceph/cluster1-client.mirror.108610.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140085.log 2024-09-16T13:50:28.992 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56272.log 2024-09-16T13:50:28.992 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134927.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97403.log 2024-09-16T13:50:28.992 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.140085.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129533.log 2024-09-16T13:50:28.993 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128255.log 2024-09-16T13:50:28.993 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.97403.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136850.log.gz 2024-09-16T13:50:28.993 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129533.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44434.log 2024-09-16T13:50:28.994 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.128255.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.128255.log.gz 2024-09-16T13:50:28.994 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80356.log 2024-09-16T13:50:28.995 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126077.log.gz 2024-09-16T13:50:28.995 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44434.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131802.log 2024-09-16T13:50:28.995 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85066.log 2024-09-16T13:50:28.995 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80356.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114310.log.gz 2024-09-16T13:50:28.997 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41804.log 2024-09-16T13:50:28.997 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.131802.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121984.log 2024-09-16T13:50:28.997 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85066.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.114390.log.gz 2024-09-16T13:50:28.997 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85066.log.gz 2024-09-16T13:50:28.997 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41804.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81098.log 2024-09-16T13:50:28.997 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.121984.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118637.log 2024-09-16T13:50:28.999 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81098.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114453.log.gz 2024-09-16T13:50:28.999 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81727.log 2024-09-16T13:50:28.999 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63583.log 2024-09-16T13:50:28.999 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118637.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93377.log 2024-09-16T13:50:28.999 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81727.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114560.log.gz 2024-09-16T13:50:29.000 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91649.log 2024-09-16T13:50:29.000 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125504.log 2024-09-16T13:50:29.001 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93377.log: /var/log/ceph/cluster1-client.mirror.91649.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127212.log.gz 2024-09-16T13:50:29.001 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109227.log 2024-09-16T13:50:29.002 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93756.log 2024-09-16T13:50:29.002 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125504.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109227.log: /var/log/ceph/cluster1-client.mirror.29715.log 2024-09-16T13:50:29.002 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91317.log 2024-09-16T13:50:29.003 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93756.log: /var/log/ceph/cluster1-client.mirror.29715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106543.log 2024-09-16T13:50:29.004 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101591.log 2024-09-16T13:50:29.004 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91317.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44644.log 2024-09-16T13:50:29.004 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106543.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.112941.log.gz 2024-09-16T13:50:29.004 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101591.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105865.log 2024-09-16T13:50:29.005 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87427.log 2024-09-16T13:50:29.005 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44644.log: /var/log/ceph/cluster1-client.mirror.105865.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28020.log 2024-09-16T13:50:29.005 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87427.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61070.log 2024-09-16T13:50:29.006 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28020.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53730.log 2024-09-16T13:50:29.006 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126372.log.gz 2024-09-16T13:50:29.007 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124606.log 2024-09-16T13:50:29.007 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61070.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136852.log 2024-09-16T13:50:29.008 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53730.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98536.log 2024-09-16T13:50:29.008 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124606.log: /var/log/ceph/cluster1-client.mirror.136852.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119767.log 2024-09-16T13:50:29.010 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.98536.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85133.log 2024-09-16T13:50:29.010 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79673.log 2024-09-16T13:50:29.010 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.119767.log: /var/log/ceph/cluster2-client.admin.85133.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55769.log 2024-09-16T13:50:29.011 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79673.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123343.log 2024-09-16T13:50:29.011 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138803.log 2024-09-16T13:50:29.012 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119565.log 2024-09-16T13:50:29.012 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123343.log: /var/log/ceph/cluster1-client.mirror.138803.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135483.log 2024-09-16T13:50:29.013 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69792.log 2024-09-16T13:50:29.013 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.119565.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67412.log 2024-09-16T13:50:29.014 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135483.log: /var/log/ceph/cluster1-client.mirror.69792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54299.log 2024-09-16T13:50:29.014 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107313.log 2024-09-16T13:50:29.015 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67412.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79452.log 2024-09-16T13:50:29.015 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54299.log: gzip /var/log/ceph/cluster2-client.mirror.107313.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61696.log 2024-09-16T13:50:29.016 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79452.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77983.log 2024-09-16T13:50:29.016 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.28175.log.gz 2024-09-16T13:50:29.016 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.61696.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37067.log 2024-09-16T13:50:29.018 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114097.log 2024-09-16T13:50:29.018 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77983.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142535.log 2024-09-16T13:50:29.018 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37067.log: /var/log/ceph/cluster1-client.mirror.114097.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43484.log 2024-09-16T13:50:29.018 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67474.log 2024-09-16T13:50:29.018 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142535.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142535.log.gz 2024-09-16T13:50:29.020 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43484.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72166.log 2024-09-16T13:50:29.020 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67474.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51490.log.gz 2024-09-16T13:50:29.020 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41089.log 2024-09-16T13:50:29.020 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144487.log 2024-09-16T13:50:29.020 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.72166.log: gzip -5 --verbose /var/log/ceph/cluster1-client.mirror.41089.log: -- /var/log/ceph/cluster1-client.mirror.46859.log 2024-09-16T13:50:29.021 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.144487.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144487.log.gzgzip 2024-09-16T13:50:29.021 INFO:teuthology.orchestra.run.smithi172.stderr:-5 --verbose -- /var/log/ceph/cluster1-client.mirror.53940.log 2024-09-16T13:50:29.021 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95522.log 2024-09-16T13:50:29.022 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46859.log: /var/log/ceph/cluster1-client.mirror.53940.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107532.log 2024-09-16T13:50:29.022 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.95522.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130432.log 2024-09-16T13:50:29.023 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126580.log 2024-09-16T13:50:29.023 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48286.log 2024-09-16T13:50:29.024 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130432.log: /var/log/ceph/cluster1-client.mirror.126580.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85077.log 2024-09-16T13:50:29.024 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.48286.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140195.log 2024-09-16T13:50:29.024 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42076.log.gz 2024-09-16T13:50:29.024 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85077.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130226.log 2024-09-16T13:50:29.025 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.140195.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69102.log 2024-09-16T13:50:29.026 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.130226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126432.log 2024-09-16T13:50:29.026 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69102.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43442.log 2024-09-16T13:50:29.027 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126432.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96746.log 2024-09-16T13:50:29.028 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43442.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63565.log.gz 2024-09-16T13:50:29.028 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29716.log 2024-09-16T13:50:29.029 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50434.log 2024-09-16T13:50:29.029 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.96746.log: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63894.log.gz 2024-09-16T13:50:29.029 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104467.log 2024-09-16T13:50:29.030 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27675.log 2024-09-16T13:50:29.030 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.50434.log: /var/log/ceph/cluster1-client.mirror.104467.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126768.log 2024-09-16T13:50:29.030 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.27675.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27675.log.gz 2024-09-16T13:50:29.031 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61851.log 2024-09-16T13:50:29.033 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126768.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64110.log.gz 2024-09-16T13:50:29.033 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68469.log 2024-09-16T13:50:29.033 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.61851.log: /var/log/ceph/cluster1-client.mirror.68469.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107171.log 2024-09-16T13:50:29.034 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44117.log.gz 2024-09-16T13:50:29.034 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39351.log 2024-09-16T13:50:29.034 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44220.log.gz 2024-09-16T13:50:29.034 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster2-client.admin.107171.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99928.log 2024-09-16T13:50:29.035 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39351.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73068.log 2024-09-16T13:50:29.035 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119355.log 2024-09-16T13:50:29.035 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99928.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.134003.log 2024-09-16T13:50:29.036 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73068.log: /var/log/ceph/cluster2-client.admin.119355.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27447.log 2024-09-16T13:50:29.036 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119355.log.gz 2024-09-16T13:50:29.036 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.134003.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136849.log 2024-09-16T13:50:29.037 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.27447.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91650.log 2024-09-16T13:50:29.037 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74732.log 2024-09-16T13:50:29.038 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136849.log: /var/log/ceph/cluster1-client.mirror.91650.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69812.log.gz 2024-09-16T13:50:29.039 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102262.log 2024-09-16T13:50:29.039 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69814.log.gz 2024-09-16T13:50:29.039 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69816.log.gz 2024-09-16T13:50:29.039 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37527.log 2024-09-16T13:50:29.040 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.74732.log: /var/log/ceph/cluster1-client.mirror.102262.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142108.log 2024-09-16T13:50:29.041 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37527.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135274.log 2024-09-16T13:50:29.041 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37527.log.gz 2024-09-16T13:50:29.041 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108724.log 2024-09-16T13:50:29.041 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142108.log: /var/log/ceph/cluster1-client.mirror.135274.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60253.log 2024-09-16T13:50:29.042 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107389.log 2024-09-16T13:50:29.042 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.108724.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46072.log.gz 2024-09-16T13:50:29.042 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41553.log 2024-09-16T13:50:29.042 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.60253.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52259.log 2024-09-16T13:50:29.043 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110735.log 2024-09-16T13:50:29.043 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41553.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25938.log 2024-09-16T13:50:29.044 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52259.log: /var/log/ceph/cluster1-client.mirror.110735.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80613.log.gz 2024-09-16T13:50:29.044 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73880.log 2024-09-16T13:50:29.044 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.25938.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40274.log 2024-09-16T13:50:29.044 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25938.log.gz 2024-09-16T13:50:29.045 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.73880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124981.log 2024-09-16T13:50:29.046 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61462.log 2024-09-16T13:50:29.047 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.40274.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79452.log.gz 2024-09-16T13:50:29.047 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124981.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90601.log 2024-09-16T13:50:29.047 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose /var/log/ceph/cluster2-client.admin.61462.log: -- /var/log/ceph/cluster1-client.mirror.64109.log 2024-09-16T13:50:29.047 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.61462.log.gz 2024-09-16T13:50:29.047 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69187.log 2024-09-16T13:50:29.048 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73153.log 2024-09-16T13:50:29.048 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64109.log: /var/log/ceph/cluster1-client.mirror.69187.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30513.log 2024-09-16T13:50:29.048 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73153.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62822.log 2024-09-16T13:50:29.048 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58167.log.gz 2024-09-16T13:50:29.051 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30513.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50024.log 2024-09-16T13:50:29.052 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108243.log 2024-09-16T13:50:29.052 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.62822.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110602.log 2024-09-16T13:50:29.052 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.50024.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.60310.log.gz 2024-09-16T13:50:29.052 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108243.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64537.log 2024-09-16T13:50:29.052 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.108243.log.gz 2024-09-16T13:50:29.052 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.admin.110602.log: -5 --verbose -- /var/log/ceph/cluster2-client.admin.72040.log 2024-09-16T13:50:29.052 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110602.log.gz 2024-09-16T13:50:29.052 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82193.log 2024-09-16T13:50:29.052 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.72040.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72040.log.gz 2024-09-16T13:50:29.055 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50232.log 2024-09-16T13:50:29.055 INFO:teuthology.orchestra.run.smithi172.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38205.log.gz 2024-09-16T13:50:29.056 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90348.log 2024-09-16T13:50:29.056 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.82193.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.82193.log.gz 2024-09-16T13:50:29.056 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.50232.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64976.log 2024-09-16T13:50:29.056 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96043.log 2024-09-16T13:50:29.056 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64976.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73994.log 2024-09-16T13:50:29.056 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99395.log 2024-09-16T13:50:29.056 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.96043.log: /var/log/ceph/cluster1-client.admin.73994.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124119.log 2024-09-16T13:50:29.056 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99395.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53121.log 2024-09-16T13:50:29.056 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.31689.log.gz 2024-09-16T13:50:29.058 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124119.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31690.log.gz 2024-09-16T13:50:29.058 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33837.log 2024-09-16T13:50:29.058 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86602.log 2024-09-16T13:50:29.059 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53121.log: /var/log/ceph/cluster1-client.mirror.33837.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111833.log 2024-09-16T13:50:29.059 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.86602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136912.log 2024-09-16T13:50:29.059 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90896.log.gz 2024-09-16T13:50:29.059 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127664.log 2024-09-16T13:50:29.060 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111833.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118929.log 2024-09-16T13:50:29.060 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43172.log 2024-09-16T13:50:29.060 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127664.log: /var/log/ceph/cluster2-client.admin.118929.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.118929.log.gz 2024-09-16T13:50:29.060 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91650.log.gz 2024-09-16T13:50:29.061 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55620.log 2024-09-16T13:50:29.061 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43172.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137481.log 2024-09-16T13:50:29.061 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142902.log 2024-09-16T13:50:29.063 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55620.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114351.log 2024-09-16T13:50:29.063 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137481.log: /var/log/ceph/cluster2-client.admin.142902.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34750.log 2024-09-16T13:50:29.063 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133416.log 2024-09-16T13:50:29.063 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53332.log 2024-09-16T13:50:29.063 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34750.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44386.log 2024-09-16T13:50:29.064 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133416.log: /var/log/ceph/cluster1-client.mirror.53332.log: 89.9%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.91946.log.gz-5 2024-09-16T13:50:29.064 INFO:teuthology.orchestra.run.smithi172.stderr:--verbose -- /var/log/ceph/cluster1-client.mirror.101123.log 2024-09-16T13:50:29.064 INFO:teuthology.orchestra.run.smithi172.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.97725.log.gz 2024-09-16T13:50:29.066 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134265.log 2024-09-16T13:50:29.066 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44386.log: /var/log/ceph/cluster1-client.mirror.101123.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43755.log 2024-09-16T13:50:29.066 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126076.log 2024-09-16T13:50:29.066 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134265.log: /var/log/ceph/cluster1-client.mirror.43755.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98773.log 2024-09-16T13:50:29.067 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64736.log 2024-09-16T13:50:29.067 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126076.log: /var/log/ceph/cluster1-client.mirror.98773.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117041.log 2024-09-16T13:50:29.067 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123720.log 2024-09-16T13:50:29.068 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144406.log 2024-09-16T13:50:29.068 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.117041.log: /var/log/ceph/cluster1-client.mirror.123720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137903.log 2024-09-16T13:50:29.069 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.144406.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106318.log 2024-09-16T13:50:29.069 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144406.log.gz 2024-09-16T13:50:29.069 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137903.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51220.log 2024-09-16T13:50:29.070 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106318.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115614.log 2024-09-16T13:50:29.070 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96308.log 2024-09-16T13:50:29.071 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51220.log: gzip -5 /var/log/ceph/cluster1-client.mirror.115614.log: --verbose -- /var/log/ceph/cluster2-client.admin.61072.log 2024-09-16T13:50:29.071 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96308.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84524.log 2024-09-16T13:50:29.072 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61072.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114518.log.gz 2024-09-16T13:50:29.073 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61499.log 2024-09-16T13:50:29.073 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.84524.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107051.log 2024-09-16T13:50:29.074 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142531.log 2024-09-16T13:50:29.074 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61499.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.61499.log.gz 2024-09-16T13:50:29.074 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28292.log 2024-09-16T13:50:29.075 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107051.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92664.log.gz 2024-09-16T13:50:29.075 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142531.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29337.log 2024-09-16T13:50:29.075 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142531.log.gz 2024-09-16T13:50:29.076 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.28292.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139331.log 2024-09-16T13:50:29.076 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77856.log.gz 2024-09-16T13:50:29.085 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29337.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91735.log 2024-09-16T13:50:29.085 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80504.log 2024-09-16T13:50:29.086 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136849.log.gz 2024-09-16T13:50:29.086 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91735.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121594.log 2024-09-16T13:50:29.086 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80504.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49661.log 2024-09-16T13:50:29.086 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.121594.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80566.log 2024-09-16T13:50:29.086 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.49661.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.139583.log.gz 2024-09-16T13:50:29.086 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73096.log 2024-09-16T13:50:29.086 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80566.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59987.log 2024-09-16T13:50:29.086 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28394.log 2024-09-16T13:50:29.086 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59987.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48713.log 2024-09-16T13:50:29.086 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121074.log 2024-09-16T13:50:29.086 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28394.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48713.log: /var/log/ceph/cluster1-client.admin.106544.log 2024-09-16T13:50:29.086 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121074.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136978.log 2024-09-16T13:50:29.086 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137903.log.gz 2024-09-16T13:50:29.087 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106544.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62266.log 2024-09-16T13:50:29.087 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123343.log.gz 2024-09-16T13:50:29.087 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118942.log 2024-09-16T13:50:29.087 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136978.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126499.log 2024-09-16T13:50:29.087 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.62266.log: /var/log/ceph/cluster2-client.admin.118942.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.118942.log.gz 2024-09-16T13:50:29.087 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97611.log 2024-09-16T13:50:29.087 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126499.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38007.log 2024-09-16T13:50:29.087 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.97611.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116180.log 2024-09-16T13:50:29.087 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.38007.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38007.log.gz 2024-09-16T13:50:29.087 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.admin.132077.log.gz 2024-09-16T13:50:29.087 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61945.log 2024-09-16T13:50:29.087 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116180.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63122.log 2024-09-16T13:50:29.087 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.61945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94357.log 2024-09-16T13:50:29.093 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81669.log 2024-09-16T13:50:29.093 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73384.log 2024-09-16T13:50:29.093 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.94357.log: /var/log/ceph/cluster1-client.mirror.81669.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59553.log 2024-09-16T13:50:29.093 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140085.log.gz 2024-09-16T13:50:29.093 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39065.log 2024-09-16T13:50:29.093 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73384.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73384.log.gz 2024-09-16T13:50:29.093 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59553.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61059.log 2024-09-16T13:50:29.093 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.39065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138280.log 2024-09-16T13:50:29.093 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61059.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103648.log 2024-09-16T13:50:29.094 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128274.log 2024-09-16T13:50:29.094 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138280.log: gzip -5 /var/log/ceph/cluster1-client.mirror.103648.log: --verbose -- /var/log/ceph/cluster1-client.mirror.75461.log 2024-09-16T13:50:29.094 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115506.log 2024-09-16T13:50:29.094 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.128274.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85945.log 2024-09-16T13:50:29.094 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75461.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126432.log.gz 2024-09-16T13:50:29.094 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115506.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143790.log 2024-09-16T13:50:29.094 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.85945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132050.log 2024-09-16T13:50:29.094 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.143790.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45383.log 2024-09-16T13:50:29.094 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119451.log 2024-09-16T13:50:29.094 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2%/var/log/ceph/cluster1-client.admin.132050.log: -- replaced with /var/log/ceph/cluster1-client.admin.143790.log.gz 2024-09-16T13:50:29.094 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78570.log 2024-09-16T13:50:29.094 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45383.log: /var/log/ceph/cluster2-client.mirror.119451.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125739.log 2024-09-16T13:50:29.094 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78570.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32151.log 2024-09-16T13:50:29.094 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29186.log 2024-09-16T13:50:29.097 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114409.log 2024-09-16T13:50:29.097 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32151.log: /var/log/ceph/cluster1-client.mirror.29186.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112563.log 2024-09-16T13:50:29.097 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114409.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102386.log 2024-09-16T13:50:29.097 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112563.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122492.log 2024-09-16T13:50:29.097 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121563.log 2024-09-16T13:50:29.097 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102386.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122492.log: /var/log/ceph/cluster1-client.mirror.111428.log 2024-09-16T13:50:29.097 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121563.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111722.log.gz 2024-09-16T13:50:29.097 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.119767.log.gz 2024-09-16T13:50:29.097 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92410.log 2024-09-16T13:50:29.097 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111428.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53225.log 2024-09-16T13:50:29.097 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92410.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26304.log 2024-09-16T13:50:29.099 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96860.log 2024-09-16T13:50:29.099 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53225.log: 19.0% -- replaced with /var/log/ceph/cluster1-client.admin.121984.log.gz 2024-09-16T13:50:29.099 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.26304.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73450.log 2024-09-16T13:50:29.099 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.26304.log.gz 2024-09-16T13:50:29.099 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.96860.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48554.log 2024-09-16T13:50:29.099 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.73450.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30469.log 2024-09-16T13:50:29.099 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.48554.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45970.log 2024-09-16T13:50:29.100 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30469.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132528.log 2024-09-16T13:50:29.101 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45970.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121157.log 2024-09-16T13:50:29.101 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.130432.log.gz 2024-09-16T13:50:29.102 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132528.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91358.log 2024-09-16T13:50:29.102 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121356.log 2024-09-16T13:50:29.102 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121157.log: /var/log/ceph/cluster1-client.mirror.91358.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102498.log 2024-09-16T13:50:29.102 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.121356.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121356.log.gz 2024-09-16T13:50:29.103 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95760.log 2024-09-16T13:50:29.103 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102498.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126768.log.gz 2024-09-16T13:50:29.103 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127214.log 2024-09-16T13:50:29.105 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115925.log 2024-09-16T13:50:29.105 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.95760.log: /var/log/ceph/cluster1-client.mirror.127214.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120518.log 2024-09-16T13:50:29.105 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41335.log 2024-09-16T13:50:29.105 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115925.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51857.log 2024-09-16T13:50:29.105 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.120518.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84028.log 2024-09-16T13:50:29.107 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41335.log: /var/log/ceph/cluster1-client.mirror.51857.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133216.log 2024-09-16T13:50:29.107 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100891.log 2024-09-16T13:50:29.107 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.84028.log: /var/log/ceph/cluster1-client.mirror.133216.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142326.log 2024-09-16T13:50:29.107 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.128274.log.gz 2024-09-16T13:50:29.107 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118938.log 2024-09-16T13:50:29.108 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100891.log: /var/log/ceph/cluster2-client.mirror.142326.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122880.log.gz 2024-09-16T13:50:29.108 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127215.log 2024-09-16T13:50:29.112 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118938.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97041.log 2024-09-16T13:50:29.112 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.118938.log.gz 2024-09-16T13:50:29.112 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53541.log 2024-09-16T13:50:29.112 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.97041.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108475.log 2024-09-16T13:50:29.112 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73144.log 2024-09-16T13:50:29.112 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53541.log: /var/log/ceph/cluster2-client.mirror.108475.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113630.log 2024-09-16T13:50:29.112 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73144.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88922.log 2024-09-16T13:50:29.112 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113630.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88280.log 2024-09-16T13:50:29.112 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88922.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26278.log 2024-09-16T13:50:29.114 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.88280.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39236.log 2024-09-16T13:50:29.115 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.26278.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79830.log 2024-09-16T13:50:29.115 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.26278.log.gz 2024-09-16T13:50:29.115 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.39236.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39236.log.gz 2024-09-16T13:50:29.115 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84376.log 2024-09-16T13:50:29.115 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79830.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28772.log 2024-09-16T13:50:29.115 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40251.log.gz 2024-09-16T13:50:29.115 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.84376.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110757.log 2024-09-16T13:50:29.116 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101944.log 2024-09-16T13:50:29.116 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.28772.log: /var/log/ceph/cluster1-client.mirror.110757.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69686.log 2024-09-16T13:50:29.116 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101944.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39459.log 2024-09-16T13:50:29.117 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69686.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29549.log.gz 2024-09-16T13:50:29.117 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96886.log 2024-09-16T13:50:29.118 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39459.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62588.log 2024-09-16T13:50:29.119 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.96886.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46864.log 2024-09-16T13:50:29.119 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131533.log 2024-09-16T13:50:29.119 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.62588.log: gzip /var/log/ceph/cluster1-client.admin.46864.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121001.log 2024-09-16T13:50:29.120 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46864.log.gz 2024-09-16T13:50:29.120 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.131533.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36290.log 2024-09-16T13:50:29.121 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121001.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134369.log 2024-09-16T13:50:29.121 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32151.log.gzgzip 2024-09-16T13:50:29.121 INFO:teuthology.orchestra.run.smithi172.stderr:-5 --verbose -- /var/log/ceph/cluster1-client.admin.120196.log 2024-09-16T13:50:29.121 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134369.log: /var/log/ceph/cluster1-client.admin.36290.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137163.log 2024-09-16T13:50:29.122 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52506.log 2024-09-16T13:50:29.122 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.120196.log: /var/log/ceph/cluster1-client.mirror.137163.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44322.log 2024-09-16T13:50:29.122 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56482.log 2024-09-16T13:50:29.123 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52506.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41335.log.gz 2024-09-16T13:50:29.123 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32241.log 2024-09-16T13:50:29.124 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44322.log: /var/log/ceph/cluster1-client.mirror.56482.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73372.log 2024-09-16T13:50:29.124 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32241.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43443.log.gz 2024-09-16T13:50:29.124 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79352.log 2024-09-16T13:50:29.125 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73372.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65268.log 2024-09-16T13:50:29.125 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73372.log.gz 2024-09-16T13:50:29.126 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79352.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43484.log.gz 2024-09-16T13:50:29.126 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37540.log 2024-09-16T13:50:29.127 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65268.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50501.log 2024-09-16T13:50:29.127 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37540.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42223.log 2024-09-16T13:50:29.127 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.50501.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129534.log 2024-09-16T13:50:29.128 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42223.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44386.log.gz 2024-09-16T13:50:29.129 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96255.log 2024-09-16T13:50:29.129 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44434.log.gz 2024-09-16T13:50:29.129 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61603.log 2024-09-16T13:50:29.130 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129534.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101652.log 2024-09-16T13:50:29.130 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96255.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96255.log.gz 2024-09-16T13:50:29.130 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.61603.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79200.log 2024-09-16T13:50:29.131 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34972.log 2024-09-16T13:50:29.131 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79200.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94396.log 2024-09-16T13:50:29.132 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.34972.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34972.log.gz 2024-09-16T13:50:29.132 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117025.log 2024-09-16T13:50:29.134 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.94396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95120.log 2024-09-16T13:50:29.134 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73719.log 2024-09-16T13:50:29.134 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117025.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.117025.log.gz 2024-09-16T13:50:29.134 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.95120.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87210.log 2024-09-16T13:50:29.134 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117115.log 2024-09-16T13:50:29.134 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.73719.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37600.log 2024-09-16T13:50:29.134 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.87210.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89644.log 2024-09-16T13:50:29.134 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.117115.log: /var/log/ceph/cluster2-client.admin.37600.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128467.log 2024-09-16T13:50:29.136 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89644.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26460.log 2024-09-16T13:50:29.136 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.128467.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27686.log 2024-09-16T13:50:29.136 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50957.log 2024-09-16T13:50:29.136 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.26460.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.26460.log.gz 2024-09-16T13:50:29.136 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.27686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80828.log 2024-09-16T13:50:29.136 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.50957.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45019.log 2024-09-16T13:50:29.138 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93755.log 2024-09-16T13:50:29.138 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80828.log: /var/log/ceph/cluster1-client.mirror.45019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100602.log 2024-09-16T13:50:29.138 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123344.log 2024-09-16T13:50:29.138 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93755.log: /var/log/ceph/cluster1-client.mirror.100602.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143957.log 2024-09-16T13:50:29.138 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123344.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68255.log 2024-09-16T13:50:29.139 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.143957.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26330.log 2024-09-16T13:50:29.140 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68255.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.143957.log.gz 2024-09-16T13:50:29.140 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104069.log 2024-09-16T13:50:29.140 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136808.log 2024-09-16T13:50:29.140 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.26330.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.26330.log.gz 2024-09-16T13:50:29.141 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104069.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42433.log 2024-09-16T13:50:29.141 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136808.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116794.log 2024-09-16T13:50:29.141 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45970.log.gz 2024-09-16T13:50:29.142 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100747.log 2024-09-16T13:50:29.142 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42433.log: /var/log/ceph/cluster1-client.mirror.116794.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54024.log 2024-09-16T13:50:29.142 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49805.log 2024-09-16T13:50:29.143 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100747.log: /var/log/ceph/cluster1-client.mirror.54024.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104177.log 2024-09-16T13:50:29.143 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53756.log 2024-09-16T13:50:29.143 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49805.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65313.log 2024-09-16T13:50:29.143 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104177.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49805.log.gz 2024-09-16T13:50:29.143 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138802.log 2024-09-16T13:50:29.144 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65313.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43023.log 2024-09-16T13:50:29.144 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57178.log 2024-09-16T13:50:29.144 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138802.log: /var/log/ceph/cluster1-client.mirror.43023.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54570.log 2024-09-16T13:50:29.145 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57178.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50688.log.gz 2024-09-16T13:50:29.145 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96128.log 2024-09-16T13:50:29.145 INFO:teuthology.orchestra.run.smithi172.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.admin.50662.log.gz 2024-09-16T13:50:29.147 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.54570.log: -5 --verbose -- /var/log/ceph/cluster2-client.admin.118955.log 2024-09-16T13:50:29.147 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.96128.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142952.log 2024-09-16T13:50:29.147 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54805.log 2024-09-16T13:50:29.147 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118955.log: /var/log/ceph/cluster2-client.admin.142952.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48875.log 2024-09-16T13:50:29.147 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142952.log.gz 2024-09-16T13:50:29.148 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54805.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38662.log.gz 2024-09-16T13:50:29.148 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63373.log 2024-09-16T13:50:29.148 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.48875.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60786.log 2024-09-16T13:50:29.151 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114349.log 2024-09-16T13:50:29.151 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63373.log: /var/log/ceph/cluster2-client.mirror.60786.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31401.log 2024-09-16T13:50:29.151 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114349.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34466.log 2024-09-16T13:50:29.151 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45168.log 2024-09-16T13:50:29.151 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31401.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46859.log.gz 2024-09-16T13:50:29.151 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34466.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30136.log 2024-09-16T13:50:29.151 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.45168.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92621.log 2024-09-16T13:50:29.151 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109066.log 2024-09-16T13:50:29.153 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92621.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69398.log 2024-09-16T13:50:29.153 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44532.log 2024-09-16T13:50:29.153 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109066.log: /var/log/ceph/cluster1-client.mirror.69398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104532.log 2024-09-16T13:50:29.153 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.44532.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89927.log 2024-09-16T13:50:29.153 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105416.log 2024-09-16T13:50:29.154 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89927.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.142108.log.gz 2024-09-16T13:50:29.154 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36835.log 2024-09-16T13:50:29.154 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105416.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70302.log 2024-09-16T13:50:29.154 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.36835.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108181.log 2024-09-16T13:50:29.155 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.70302.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.70302.log.gz 2024-09-16T13:50:29.155 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68973.log 2024-09-16T13:50:29.156 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108181.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46798.log 2024-09-16T13:50:29.157 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88573.log 2024-09-16T13:50:29.157 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68973.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120337.log /var/log/ceph/cluster1-client.admin.46798.log: 2024-09-16T13:50:29.157 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46798.log.gz 2024-09-16T13:50:29.158 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88573.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48559.log 2024-09-16T13:50:29.158 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.120337.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72542.log 2024-09-16T13:50:29.159 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.48559.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49890.log 2024-09-16T13:50:29.160 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.72542.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30980.log 2024-09-16T13:50:29.161 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.49890.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143247.log 2024-09-16T13:50:29.161 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73792.log 2024-09-16T13:50:29.161 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30980.log: /var/log/ceph/cluster1-client.admin.143247.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101656.log 2024-09-16T13:50:29.162 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.73792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45230.log 2024-09-16T13:50:29.162 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101656.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110652.log 2024-09-16T13:50:29.163 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69734.log.gz 2024-09-16T13:50:29.163 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45230.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67919.log 2024-09-16T13:50:29.163 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.143247.log.gz 2024-09-16T13:50:29.164 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74876.log 2024-09-16T13:50:29.164 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92324.log 2024-09-16T13:50:29.165 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.74876.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34254.log 2024-09-16T13:50:29.167 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92324.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68423.log 2024-09-16T13:50:29.167 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52259.log.gz 2024-09-16T13:50:29.167 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55431.log 2024-09-16T13:50:29.167 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34254.log: /var/log/ceph/cluster1-client.mirror.68423.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68422.log 2024-09-16T13:50:29.167 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.55431.log: -5 --verbose -- /var/log/ceph/cluster2-client.admin.85438.log 2024-09-16T13:50:29.167 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134031.log 2024-09-16T13:50:29.167 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68422.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126874.log 2024-09-16T13:50:29.169 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85438.log: /var/log/ceph/cluster1-client.mirror.134031.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61062.log 2024-09-16T13:50:29.169 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30348.log 2024-09-16T13:50:29.169 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126874.log: /var/log/ceph/cluster2-client.admin.61062.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56482.log.gz 2024-09-16T13:50:29.169 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98809.log 2024-09-16T13:50:29.169 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130352.log 2024-09-16T13:50:29.172 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61212.log 2024-09-16T13:50:29.173 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98809.log: /var/log/ceph/cluster2-client.admin.130352.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124268.log 2024-09-16T13:50:29.173 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130352.log.gz 2024-09-16T13:50:29.173 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61212.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74061.log 2024-09-16T13:50:29.173 INFO:teuthology.orchestra.run.smithi172.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50232.log.gz 2024-09-16T13:50:29.173 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124268.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104386.log 2024-09-16T13:50:29.173 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.74061.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49125.log 2024-09-16T13:50:29.173 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74818.log 2024-09-16T13:50:29.173 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104386.log: /var/log/ceph/cluster2-client.admin.49125.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102364.log 2024-09-16T13:50:29.173 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49125.log.gz 2024-09-16T13:50:29.173 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.74818.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120041.log 2024-09-16T13:50:29.173 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102364.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104888.log 2024-09-16T13:50:29.176 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.120041.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98219.log 2024-09-16T13:50:29.177 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93756.log.gz 2024-09-16T13:50:29.177 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104888.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41130.log 2024-09-16T13:50:29.177 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69606.log 2024-09-16T13:50:29.177 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.98219.log: /var/log/ceph/cluster1-client.mirror.41130.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32242.log 2024-09-16T13:50:29.177 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62854.log /var/log/ceph/cluster1-client.mirror.69606.log: 2024-09-16T13:50:29.177 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117980.log 2024-09-16T13:50:29.177 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32242.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79830.log.gz 2024-09-16T13:50:29.177 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.62854.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60311.log 2024-09-16T13:50:29.177 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117980.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122181.log 2024-09-16T13:50:29.177 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109831.log 2024-09-16T13:50:29.177 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.60311.log: /var/log/ceph/cluster1-client.admin.122181.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77251.log 2024-09-16T13:50:29.179 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115884.log 2024-09-16T13:50:29.179 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109831.log: /var/log/ceph/cluster1-client.mirror.77251.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91272.log 2024-09-16T13:50:29.179 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75283.log 2024-09-16T13:50:29.179 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115884.log: /var/log/ceph/cluster1-client.mirror.91272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68846.log 2024-09-16T13:50:29.180 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44914.log 2024-09-16T13:50:29.180 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68846.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58684.log 2024-09-16T13:50:29.181 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115570.log 2024-09-16T13:50:29.181 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44914.log: /var/log/ceph/cluster1-client.mirror.58684.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93084.log 2024-09-16T13:50:29.181 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115570.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30220.log 2024-09-16T13:50:29.182 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121643.log 2024-09-16T13:50:29.182 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30220.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69941.log 2024-09-16T13:50:29.182 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.107051.log.gz 2024-09-16T13:50:29.182 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127664.log.gz 2024-09-16T13:50:29.182 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121643.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97895.log 2024-09-16T13:50:29.184 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127129.log 2024-09-16T13:50:29.184 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.97895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127130.log 2024-09-16T13:50:29.184 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127129.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51626.log 2024-09-16T13:50:29.184 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99841.log 2024-09-16T13:50:29.184 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127130.log: /var/log/ceph/cluster1-client.admin.51626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95115.log 2024-09-16T13:50:29.185 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99841.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126243.log 2024-09-16T13:50:29.185 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.95115.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102640.log 2024-09-16T13:50:29.185 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.73994.log.gz 2024-09-16T13:50:29.186 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126243.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125316.log 2024-09-16T13:50:29.186 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46665.log 2024-09-16T13:50:29.186 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125316.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129537.log 2024-09-16T13:50:29.187 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100891.log.gz 2024-09-16T13:50:29.187 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46665.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29057.log 2024-09-16T13:50:29.188 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123974.log 2024-09-16T13:50:29.188 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129537.log: /var/log/ceph/cluster1-client.mirror.29057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93252.log 2024-09-16T13:50:29.189 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 /var/log/ceph/cluster1-client.mirror.123974.log: --verbose -- /var/log/ceph/cluster1-client.mirror.86826.log 2024-09-16T13:50:29.189 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129400.log 2024-09-16T13:50:29.189 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93252.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63349.log 2024-09-16T13:50:29.190 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.86826.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37275.log 2024-09-16T13:50:29.190 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129400.log: /var/log/ceph/cluster1-client.mirror.63349.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111934.log.gz 2024-09-16T13:50:29.191 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72093.log 2024-09-16T13:50:29.192 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84213.log 2024-09-16T13:50:29.192 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.37275.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85061.log 2024-09-16T13:50:29.192 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.72093.log: /var/log/ceph/cluster1-client.admin.84213.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130057.log 2024-09-16T13:50:29.192 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85061.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85061.log.gz 2024-09-16T13:50:29.193 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95798.log 2024-09-16T13:50:29.193 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92113.log 2024-09-16T13:50:29.193 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.95798.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88278.log 2024-09-16T13:50:29.195 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92113.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29058.log 2024-09-16T13:50:29.195 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.86602.log.gz 2024-09-16T13:50:29.195 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.88278.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86447.log 2024-09-16T13:50:29.195 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.88278.log.gz 2024-09-16T13:50:29.195 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120021.log 2024-09-16T13:50:29.196 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29058.log: /var/log/ceph/cluster2-client.mirror.86447.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97927.log 2024-09-16T13:50:29.196 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.120021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33502.log 2024-09-16T13:50:29.198 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.97927.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42182.log 2024-09-16T13:50:29.198 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33502.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81035.log 2024-09-16T13:50:29.198 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42434.log 2024-09-16T13:50:29.198 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42182.log: gzip -5 --verbose /var/log/ceph/cluster1-client.mirror.81035.log: -- /var/log/ceph/cluster2-client.mirror.73584.log 2024-09-16T13:50:29.200 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42434.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139163.log 2024-09-16T13:50:29.200 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.73584.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41697.log 2024-09-16T13:50:29.200 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94357.log.gz 2024-09-16T13:50:29.200 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94396.log.gz 2024-09-16T13:50:29.200 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139163.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125107.log 2024-09-16T13:50:29.201 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81037.log 2024-09-16T13:50:29.201 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41697.log: /var/log/ceph/cluster1-client.mirror.125107.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72950.log 2024-09-16T13:50:29.201 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101944.log.gz 2024-09-16T13:50:29.202 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81037.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117982.log 2024-09-16T13:50:29.202 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.72950.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91988.log 2024-09-16T13:50:29.203 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117982.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110857.log 2024-09-16T13:50:29.203 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91988.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138111.log 2024-09-16T13:50:29.203 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110857.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89927.log.gz 2024-09-16T13:50:29.205 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131121.log 2024-09-16T13:50:29.205 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129938.log 2024-09-16T13:50:29.205 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.131121.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37041.log 2024-09-16T13:50:29.205 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131121.log.gz 2024-09-16T13:50:29.206 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.129938.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38682.log 2024-09-16T13:50:29.206 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37041.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111192.log 2024-09-16T13:50:29.206 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36291.log 2024-09-16T13:50:29.206 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.38682.log: /var/log/ceph/cluster1-client.mirror.111192.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89485.log.gz 2024-09-16T13:50:29.207 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29592.log 2024-09-16T13:50:29.207 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36291.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76138.log 2024-09-16T13:50:29.208 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29592.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27164.log 2024-09-16T13:50:29.208 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55851.log 2024-09-16T13:50:29.208 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93125.log 2024-09-16T13:50:29.210 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.27164.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27333.log 2024-09-16T13:50:29.210 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55851.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.108885.log.gz 2024-09-16T13:50:29.210 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93125.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113796.log 2024-09-16T13:50:29.211 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114099.log 2024-09-16T13:50:29.211 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.27333.log: /var/log/ceph/cluster1-client.mirror.113796.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55349.log 2024-09-16T13:50:29.211 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114099.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88018.log 2024-09-16T13:50:29.212 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55349.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.120196.log.gz 2024-09-16T13:50:29.212 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78721.log 2024-09-16T13:50:29.212 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71275.log 2024-09-16T13:50:29.213 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88018.log: /var/log/ceph/cluster1-client.mirror.78721.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51088.log 2024-09-16T13:50:29.213 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73078.log 2024-09-16T13:50:29.213 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71275.log: /var/log/ceph/cluster1-client.mirror.51088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127132.log 2024-09-16T13:50:29.214 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67999.log 2024-09-16T13:50:29.215 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73078.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80146.log 2024-09-16T13:50:29.215 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127132.log: /var/log/ceph/cluster1-client.mirror.67999.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104488.log 2024-09-16T13:50:29.215 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118197.log 2024-09-16T13:50:29.215 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80146.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.118637.log.gz 2024-09-16T13:50:29.215 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104488.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122967.log 2024-09-16T13:50:29.216 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118197.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120847.log 2024-09-16T13:50:29.217 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122967.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112672.log 2024-09-16T13:50:29.217 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133450.log 2024-09-16T13:50:29.217 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.120847.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74195.log 2024-09-16T13:50:29.221 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112672.log: /var/log/ceph/cluster1-client.mirror.133450.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103758.log 2024-09-16T13:50:29.221 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.74195.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103337.log 2024-09-16T13:50:29.221 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121123.log.gz 2024-09-16T13:50:29.221 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46493.log 2024-09-16T13:50:29.221 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103758.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143884.log 2024-09-16T13:50:29.222 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103337.log: /var/log/ceph/cluster1-client.mirror.46493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66994.log 2024-09-16T13:50:29.222 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80715.log 2024-09-16T13:50:29.222 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.143884.log: 18.5% -- replaced with /var/log/ceph/cluster1-client.admin.121594.log.gz 2024-09-16T13:50:29.222 INFO:teuthology.orchestra.run.smithi172.stderr: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.143884.log.gz 2024-09-16T13:50:29.222 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66994.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72823.log 2024-09-16T13:50:29.222 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80715.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95527.log 2024-09-16T13:50:29.222 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125191.log 2024-09-16T13:50:29.222 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.72823.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111723.log 2024-09-16T13:50:29.222 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.95527.log: /var/log/ceph/cluster1-client.mirror.125191.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121643.log.gz 2024-09-16T13:50:29.223 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.129400.log.gz 2024-09-16T13:50:29.223 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139375.log 2024-09-16T13:50:29.224 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30012.log 2024-09-16T13:50:29.224 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90535.log 2024-09-16T13:50:29.224 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119398.log 2024-09-16T13:50:29.224 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30012.log: /var/log/ceph/cluster1-client.mirror.90535.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54510.log 2024-09-16T13:50:29.225 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.119398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49435.log 2024-09-16T13:50:29.225 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54510.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73376.log 2024-09-16T13:50:29.226 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.49435.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26174.log 2024-09-16T13:50:29.226 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139669.log 2024-09-16T13:50:29.226 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73376.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73376.log.gz 2024-09-16T13:50:29.226 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.26174.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.26174.log.gz 2024-09-16T13:50:29.227 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139861.log 2024-09-16T13:50:29.227 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103756.log 2024-09-16T13:50:29.228 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139669.log: /var/log/ceph/cluster1-client.mirror.139861.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77414.log 2024-09-16T13:50:29.228 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116180.log.gz 2024-09-16T13:50:29.228 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103756.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50822.log 2024-09-16T13:50:29.229 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77414.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87047.log 2024-09-16T13:50:29.229 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.50822.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71737.log 2024-09-16T13:50:29.229 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107740.log 2024-09-16T13:50:29.231 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.87047.log: gzip /var/log/ceph/cluster1-client.admin.71737.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125820.log 2024-09-16T13:50:29.232 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76218.log 2024-09-16T13:50:29.232 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107740.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.107171.log.gz 2024-09-16T13:50:29.232 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94280.log 2024-09-16T13:50:29.232 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45279.log 2024-09-16T13:50:29.232 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76218.log: /var/log/ceph/cluster1-client.admin.94280.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.94280.log.gz 2024-09-16T13:50:29.232 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143501.log 2024-09-16T13:50:29.232 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45279.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109253.log 2024-09-16T13:50:29.234 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.143501.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46332.log 2024-09-16T13:50:29.234 INFO:teuthology.orchestra.run.smithi172.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.143501.log.gz 2024-09-16T13:50:29.234 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93210.log.gz 2024-09-16T13:50:29.234 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.109253.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93167.log.gz 2024-09-16T13:50:29.234 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136743.log 2024-09-16T13:50:29.234 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46332.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132009.log 2024-09-16T13:50:29.234 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101243.log 2024-09-16T13:50:29.236 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.132009.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125318.log 2024-09-16T13:50:29.236 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101243.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113654.log 2024-09-16T13:50:29.236 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93377.log.gz 2024-09-16T13:50:29.236 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125318.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73076.log 2024-09-16T13:50:29.236 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72168.log 2024-09-16T13:50:29.237 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113654.log: /var/log/ceph/cluster2-client.admin.73076.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53961.log 2024-09-16T13:50:29.237 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.72168.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120363.log 2024-09-16T13:50:29.237 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93504.log.gz 2024-09-16T13:50:29.238 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53961.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37253.log 2024-09-16T13:50:29.238 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70477.log 2024-09-16T13:50:29.239 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.120363.log: /var/log/ceph/cluster2-client.mirror.37253.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120597.log 2024-09-16T13:50:29.239 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99502.log 2024-09-16T13:50:29.239 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.70477.log: /var/log/ceph/cluster1-client.admin.120597.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122626.log 2024-09-16T13:50:29.240 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109949.log 2024-09-16T13:50:29.240 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.99502.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.99502.log.gz 2024-09-16T13:50:29.240 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85831.log 2024-09-16T13:50:29.241 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.109949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40496.log 2024-09-16T13:50:29.241 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.85831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40534.log 2024-09-16T13:50:29.241 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40496.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112462.log 2024-09-16T13:50:29.241 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.40534.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116750.log 2024-09-16T13:50:29.242 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112462.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.129533.log.gz 2024-09-16T13:50:29.242 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35100.log 2024-09-16T13:50:29.242 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114287.log 2024-09-16T13:50:29.243 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116750.log: /var/log/ceph/cluster1-client.mirror.35100.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49824.log 2024-09-16T13:50:29.243 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72722.log 2024-09-16T13:50:29.243 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114287.log: /var/log/ceph/cluster2-client.admin.49824.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49824.log.gz 2024-09-16T13:50:29.243 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82088.log 2024-09-16T13:50:29.244 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.72722.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125950.log 2024-09-16T13:50:29.245 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93294.log 2024-09-16T13:50:29.245 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.82088.log: /var/log/ceph/cluster1-client.mirror.125950.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28846.log 2024-09-16T13:50:29.247 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142536.log 2024-09-16T13:50:29.247 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39159.log 2024-09-16T13:50:29.247 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28846.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58639.log 2024-09-16T13:50:29.247 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142536.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142536.log.gz 2024-09-16T13:50:29.247 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39159.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65374.log 2024-09-16T13:50:29.247 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.58639.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125926.log 2024-09-16T13:50:29.247 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.58639.log.gz 2024-09-16T13:50:29.247 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113822.log 2024-09-16T13:50:29.247 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65374.log: /var/log/ceph/cluster1-client.mirror.125926.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67578.log 2024-09-16T13:50:29.249 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117983.log 2024-09-16T13:50:29.249 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113822.log: /var/log/ceph/cluster1-client.mirror.67578.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112041.log 2024-09-16T13:50:29.249 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117983.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38065.log 2024-09-16T13:50:29.249 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112041.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48883.log 2024-09-16T13:50:29.250 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.38065.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85234.log 2024-09-16T13:50:29.250 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.48883.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71505.log 2024-09-16T13:50:29.250 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85234.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74450.log 2024-09-16T13:50:29.253 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71505.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119059.log 2024-09-16T13:50:29.253 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138802.log.gz 2024-09-16T13:50:29.253 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.74450.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128376.log 2024-09-16T13:50:29.253 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.119059.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88168.log 2024-09-16T13:50:29.253 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.128376.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82419.log 2024-09-16T13:50:29.253 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.128376.log.gz 2024-09-16T13:50:29.254 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113150.log 2024-09-16T13:50:29.254 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.120021.log.gz 2024-09-16T13:50:29.255 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.82419.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98732.log 2024-09-16T13:50:29.255 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113150.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94359.log 2024-09-16T13:50:29.255 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.98732.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100811.log 2024-09-16T13:50:29.255 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.94359.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.94359.log.gz 2024-09-16T13:50:29.255 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55704.log 2024-09-16T13:50:29.256 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100811.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51451.log 2024-09-16T13:50:29.256 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34042.log 2024-09-16T13:50:29.256 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55704.log: /var/log/ceph/cluster1-client.admin.51451.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99706.log 2024-09-16T13:50:29.257 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34042.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58269.log 2024-09-16T13:50:29.258 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99706.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53225.log.gz 2024-09-16T13:50:29.258 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91020.log 2024-09-16T13:50:29.259 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58269.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85857.log 2024-09-16T13:50:29.259 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91020.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88448.log 2024-09-16T13:50:29.259 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68779.log 2024-09-16T13:50:29.259 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.85857.log: /var/log/ceph/cluster1-client.mirror.88448.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48120.log 2024-09-16T13:50:29.259 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster2-client.admin.37548.log.gz 2024-09-16T13:50:29.260 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68779.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81621.log 2024-09-16T13:50:29.260 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.48120.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139623.log 2024-09-16T13:50:29.261 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81621.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71817.log 2024-09-16T13:50:29.262 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110653.log 2024-09-16T13:50:29.262 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139623.log: /var/log/ceph/cluster2-client.admin.71817.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33200.log 2024-09-16T13:50:29.263 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80825.log 2024-09-16T13:50:29.263 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110653.log: /var/log/ceph/cluster1-client.mirror.33200.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134189.log 2024-09-16T13:50:29.264 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80825.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43233.log.gz 2024-09-16T13:50:29.264 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43232.log.gz 2024-09-16T13:50:29.264 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34555.log 2024-09-16T13:50:29.264 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57599.log 2024-09-16T13:50:29.265 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134189.log: /var/log/ceph/cluster1-client.mirror.34555.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64738.log 2024-09-16T13:50:29.265 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57599.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81833.log 2024-09-16T13:50:29.266 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64738.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.140146.log 2024-09-16T13:50:29.267 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28283.log 2024-09-16T13:50:29.268 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111026.log 2024-09-16T13:50:29.268 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.140146.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.140146.log.gz 2024-09-16T13:50:29.268 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.28283.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127132.log.gz 2024-09-16T13:50:29.268 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127130.log.gz 2024-09-16T13:50:29.268 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117158.log 2024-09-16T13:50:29.268 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.111026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43272.log 2024-09-16T13:50:29.271 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.117158.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40188.log 2024-09-16T13:50:29.271 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.132426.log.gz 2024-09-16T13:50:29.271 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125024.log 2024-09-16T13:50:29.272 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43272.log: /var/log/ceph/cluster1-client.mirror.40188.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64238.log 2024-09-16T13:50:29.272 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125024.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92535.log 2024-09-16T13:50:29.272 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64238.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121644.log 2024-09-16T13:50:29.272 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115614.log.gz 2024-09-16T13:50:29.272 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.92535.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124269.log 2024-09-16T13:50:29.272 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.121644.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65057.log 2024-09-16T13:50:29.272 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69582.log 2024-09-16T13:50:29.272 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137019.log 2024-09-16T13:50:29.274 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69582.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58831.log 2024-09-16T13:50:29.274 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70325.log 2024-09-16T13:50:29.274 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137440.log 2024-09-16T13:50:29.274 INFO:teuthology.orchestra.run.smithi172.stderr: 67.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50501.log.gz 2024-09-16T13:50:29.275 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.70325.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.70325.log.gz 2024-09-16T13:50:29.275 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78089.log 2024-09-16T13:50:29.275 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137440.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121442.log 2024-09-16T13:50:29.275 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114729.log 2024-09-16T13:50:29.276 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121442.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38013.log 2024-09-16T13:50:29.276 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114729.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50118.log 2024-09-16T13:50:29.276 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.38013.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38013.log.gz 2024-09-16T13:50:29.276 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.28394.log.gz 2024-09-16T13:50:29.276 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130817.log 2024-09-16T13:50:29.278 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.50118.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100958.log 2024-09-16T13:50:29.279 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62038.log 2024-09-16T13:50:29.279 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.130817.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65732.log 2024-09-16T13:50:29.279 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100958.log: /var/log/ceph/cluster2-client.mirror.62038.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100411.log 2024-09-16T13:50:29.279 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65732.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121839.log 2024-09-16T13:50:29.279 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121022.log 2024-09-16T13:50:29.280 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100411.log: /var/log/ceph/cluster1-client.mirror.121839.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32241.log.gz 2024-09-16T13:50:29.280 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89006.log 2024-09-16T13:50:29.280 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56188.log 2024-09-16T13:50:29.281 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.121022.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121022.log.gz 2024-09-16T13:50:29.281 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89006.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107537.log 2024-09-16T13:50:29.282 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77730.log 2024-09-16T13:50:29.282 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56188.log: /var/log/ceph/cluster2-client.admin.107537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34423.log 2024-09-16T13:50:29.282 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.28020.log.gz 2024-09-16T13:50:29.282 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77730.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75608.log 2024-09-16T13:50:29.282 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30304.log 2024-09-16T13:50:29.284 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34423.log: /var/log/ceph/cluster1-client.mirror.75608.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50521.log 2024-09-16T13:50:29.284 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30304.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65794.log 2024-09-16T13:50:29.284 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143133.log 2024-09-16T13:50:29.284 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.50521.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70195.log 2024-09-16T13:50:29.285 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65794.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27108.log 2024-09-16T13:50:29.287 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.143133.log: /var/log/ceph/cluster1-client.mirror.70195.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96354.log 2024-09-16T13:50:29.287 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster2-client.mirror.27108.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63101.log 2024-09-16T13:50:29.287 INFO:teuthology.orchestra.run.smithi172.stderr: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.25669.log.gz 2024-09-16T13:50:29.287 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34944.log 2024-09-16T13:50:29.287 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96354.log: /var/log/ceph/cluster1-client.mirror.63101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93001.log 2024-09-16T13:50:29.287 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122836.log 2024-09-16T13:50:29.287 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.34944.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34944.log.gz 2024-09-16T13:50:29.287 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93001.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140094.log 2024-09-16T13:50:29.287 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81519.log 2024-09-16T13:50:29.289 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122836.log: /var/log/ceph/cluster1-client.mirror.140094.log: gzip 91.7%-5 --verbose -- replaced with /var/log/ceph/cluster1-client.admin.143133.log.gz-- 2024-09-16T13:50:29.290 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.74645.log 2024-09-16T13:50:29.290 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133165.log 2024-09-16T13:50:29.290 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81519.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.27447.log.gz 2024-09-16T13:50:29.290 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.74645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39982.log 2024-09-16T13:50:29.290 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91315.log 2024-09-16T13:50:29.290 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.39982.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39982.log.gz 2024-09-16T13:50:29.290 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127085.log 2024-09-16T13:50:29.290 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91315.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122158.log 2024-09-16T13:50:29.291 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127085.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95113.log 2024-09-16T13:50:29.292 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122158.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124185.log 2024-09-16T13:50:29.292 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.95113.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133887.log 2024-09-16T13:50:29.292 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69161.log 2024-09-16T13:50:29.293 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133887.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124773.log 2024-09-16T13:50:29.293 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69161.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81247.log 2024-09-16T13:50:29.293 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63373.log.gz 2024-09-16T13:50:29.293 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59990.log 2024-09-16T13:50:29.295 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124773.log: /var/log/ceph/cluster1-client.mirror.81247.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76590.log 2024-09-16T13:50:29.295 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59990.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130995.log 2024-09-16T13:50:29.295 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76590.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131393.log 2024-09-16T13:50:29.295 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130995.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80614.log 2024-09-16T13:50:29.296 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.131393.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41130.log.gz 2024-09-16T13:50:29.296 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47899.log 2024-09-16T13:50:29.297 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80614.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130210.log 2024-09-16T13:50:29.297 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.47899.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98210.log 2024-09-16T13:50:29.297 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.130210.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68552.log 2024-09-16T13:50:29.298 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.98210.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.21311.log 2024-09-16T13:50:29.299 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45279.log.gz 2024-09-16T13:50:29.299 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68552.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144091.log 2024-09-16T13:50:29.299 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.21311.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.21311.log.gz 2024-09-16T13:50:29.299 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117005.log 2024-09-16T13:50:29.300 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.144091.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68845.log.gz 2024-09-16T13:50:29.301 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48547.log 2024-09-16T13:50:29.301 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117005.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137818.log 2024-09-16T13:50:29.301 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.117005.log.gz 2024-09-16T13:50:29.301 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.48547.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52191.log 2024-09-16T13:50:29.303 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137818.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68951.log.gz 2024-09-16T13:50:29.303 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119793.log 2024-09-16T13:50:29.303 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52191.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103967.log 2024-09-16T13:50:29.303 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142635.log 2024-09-16T13:50:29.303 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.119793.log: /var/log/ceph/cluster1-client.mirror.103967.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67077.log 2024-09-16T13:50:29.304 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142635.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56692.log.gz 2024-09-16T13:50:29.304 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73739.log 2024-09-16T13:50:29.305 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54173.log 2024-09-16T13:50:29.305 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67077.log: /var/log/ceph/cluster1-client.admin.73739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50976.log 2024-09-16T13:50:29.305 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54173.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103481.log 2024-09-16T13:50:29.305 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.144091.log.gz 2024-09-16T13:50:29.306 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.50976.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96285.log 2024-09-16T13:50:29.307 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103481.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141872.log 2024-09-16T13:50:29.307 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96285.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.140881.log 2024-09-16T13:50:29.307 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.141872.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68779.log.gz 2024-09-16T13:50:29.308 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75086.log 2024-09-16T13:50:29.308 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.140881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124394.log 2024-09-16T13:50:29.308 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75086.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53668.log 2024-09-16T13:50:29.312 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124394.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57666.log 2024-09-16T13:50:29.312 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53668.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66994.log.gz 2024-09-16T13:50:29.312 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45863.log 2024-09-16T13:50:29.312 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57666.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138028.log 2024-09-16T13:50:29.312 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45863.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31099.log 2024-09-16T13:50:29.312 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138028.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107632.log 2024-09-16T13:50:29.312 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65479.log 2024-09-16T13:50:29.312 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68422.log.gz 2024-09-16T13:50:29.313 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31099.log: /var/log/ceph/cluster2-client.admin.107632.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93041.log 2024-09-16T13:50:29.313 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65479.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109988.log 2024-09-16T13:50:29.315 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93041.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48305.log 2024-09-16T13:50:29.315 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109988.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74605.log 2024-09-16T13:50:29.315 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54865.log.gz 2024-09-16T13:50:29.316 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54805.log.gz 2024-09-16T13:50:29.316 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.48305.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81458.log 2024-09-16T13:50:29.316 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.74605.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139013.log 2024-09-16T13:50:29.316 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54928.log.gz 2024-09-16T13:50:29.316 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34424.log 2024-09-16T13:50:29.316 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81458.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41376.log 2024-09-16T13:50:29.316 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139013.log: /var/log/ceph/cluster1-client.mirror.34424.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135189.log 2024-09-16T13:50:29.316 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41376.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103969.log 2024-09-16T13:50:29.319 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135189.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65605.log.gz 2024-09-16T13:50:29.319 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51087.log 2024-09-16T13:50:29.319 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103969.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134579.log 2024-09-16T13:50:29.319 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51087.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76676.log 2024-09-16T13:50:29.320 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134579.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77042.log 2024-09-16T13:50:29.320 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69582.log.gz 2024-09-16T13:50:29.320 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69524.log.gz 2024-09-16T13:50:29.320 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128424.log 2024-09-16T13:50:29.320 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76676.log: /var/log/ceph/cluster1-client.mirror.77042.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73906.log 2024-09-16T13:50:29.320 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.128424.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40224.log 2024-09-16T13:50:29.320 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.73906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30680.log 2024-09-16T13:50:29.320 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73054.log 2024-09-16T13:50:29.323 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40224.log: /var/log/ceph/cluster1-client.mirror.30680.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97875.log 2024-09-16T13:50:29.323 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73054.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73054.log.gz 2024-09-16T13:50:29.323 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29592.log.gz 2024-09-16T13:50:29.323 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102890.log 2024-09-16T13:50:29.323 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.97875.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54172.log 2024-09-16T13:50:29.323 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.102890.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39843.log 2024-09-16T13:50:29.324 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101862.log 2024-09-16T13:50:29.324 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54172.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132337.log 2024-09-16T13:50:29.324 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39843.log: /var/log/ceph/cluster1-client.mirror.101862.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101444.log 2024-09-16T13:50:29.324 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.132337.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93168.log 2024-09-16T13:50:29.324 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101444.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111997.log 2024-09-16T13:50:29.324 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32869.log 2024-09-16T13:50:29.325 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111997.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55115.log 2024-09-16T13:50:29.325 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99952.log 2024-09-16T13:50:29.325 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32869.log: /var/log/ceph/cluster1-client.mirror.55115.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86468.log 2024-09-16T13:50:29.326 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.75283.log.gz 2024-09-16T13:50:29.326 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99952.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84734.log 2024-09-16T13:50:29.326 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.86468.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42751.log 2024-09-16T13:50:29.327 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.73792.log.gz 2024-09-16T13:50:29.327 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.84734.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61268.log 2024-09-16T13:50:29.327 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63298.log 2024-09-16T13:50:29.327 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42751.log: /var/log/ceph/cluster2-client.admin.61268.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79514.log 2024-09-16T13:50:29.327 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.63298.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63298.log.gz 2024-09-16T13:50:29.327 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128993.log 2024-09-16T13:50:29.327 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.107537.log.gz 2024-09-16T13:50:29.329 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137860.log 2024-09-16T13:50:29.329 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.128993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28238.log 2024-09-16T13:50:29.329 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78361.log 2024-09-16T13:50:29.329 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137860.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92536.log 2024-09-16T13:50:29.329 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28238.log: /var/log/ceph/cluster1-client.mirror.78361.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108155.log 2024-09-16T13:50:29.331 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103732.log 2024-09-16T13:50:29.331 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108155.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80828.log.gz 2024-09-16T13:50:29.331 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106920.log 2024-09-16T13:50:29.331 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103732.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49437.log 2024-09-16T13:50:29.331 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98595.log 2024-09-16T13:50:29.331 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106920.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30724.log 2024-09-16T13:50:29.336 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.49437.log: /var/log/ceph/cluster1-client.mirror.98595.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70303.log 2024-09-16T13:50:29.336 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30724.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88168.log.gz 2024-09-16T13:50:29.336 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79140.log 2024-09-16T13:50:29.336 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.70303.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74496.log 2024-09-16T13:50:29.336 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.70303.log.gz 2024-09-16T13:50:29.336 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.87047.log.gz 2024-09-16T13:50:29.336 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88328.log 2024-09-16T13:50:29.336 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79140.log: /var/log/ceph/cluster2-client.mirror.74496.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110085.log 2024-09-16T13:50:29.336 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88328.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86219.log 2024-09-16T13:50:29.337 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.110085.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86784.log 2024-09-16T13:50:29.337 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117981.log 2024-09-16T13:50:29.340 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.86219.log: /var/log/ceph/cluster1-client.admin.86784.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142009.log 2024-09-16T13:50:29.340 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130403.log 2024-09-16T13:50:29.341 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117981.log: /var/log/ceph/cluster2-client.admin.142009.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102303.log 2024-09-16T13:50:29.341 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.95983.log 2024-09-16T13:50:29.341 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130403.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28247.log 2024-09-16T13:50:29.341 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27540.log 2024-09-16T13:50:29.341 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.95983.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42434.log.gz 2024-09-16T13:50:29.341 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28247.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133821.log 2024-09-16T13:50:29.341 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.27540.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27540.log.gz 2024-09-16T13:50:29.341 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36801.log 2024-09-16T13:50:29.341 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26706.log 2024-09-16T13:50:29.341 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54236.log.gz 2024-09-16T13:50:29.341 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61676.log 2024-09-16T13:50:29.341 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36801.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87024.log 2024-09-16T13:50:29.341 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.26706.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.26706.log.gz 2024-09-16T13:50:29.342 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.27126.log 2024-09-16T13:50:29.342 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.61676.log: /var/log/ceph/cluster1-client.mirror.87024.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34812.log 2024-09-16T13:50:29.342 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.0.27126.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39004.log 2024-09-16T13:50:29.342 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34812.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77565.log 2024-09-16T13:50:29.347 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.39004.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44533.log 2024-09-16T13:50:29.347 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77565.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113929.log 2024-09-16T13:50:29.347 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose /var/log/ceph/cluster1-client.mirror.44533.log: -- /var/log/ceph/cluster2-client.mirror.27040.log 2024-09-16T13:50:29.347 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51770.log 2024-09-16T13:50:29.347 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113929.log: /var/log/ceph/cluster2-client.mirror.27040.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116496.log 2024-09-16T13:50:29.348 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.51770.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130857.log 2024-09-16T13:50:29.348 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51770.log.gz 2024-09-16T13:50:29.348 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99242.log 2024-09-16T13:50:29.348 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116496.log: /var/log/ceph/cluster2-client.mirror.130857.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139288.log 2024-09-16T13:50:29.348 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68973.log.gz 2024-09-16T13:50:29.348 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99242.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36836.log 2024-09-16T13:50:29.348 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139288.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121769.log 2024-09-16T13:50:29.348 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93884.log 2024-09-16T13:50:29.348 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.36836.log: /var/log/ceph/cluster1-client.admin.121769.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121769.log.gz 2024-09-16T13:50:29.348 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69161.log.gz 2024-09-16T13:50:29.348 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136769.log 2024-09-16T13:50:29.348 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93884.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123279.log 2024-09-16T13:50:29.349 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97020.log 2024-09-16T13:50:29.349 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136769.log: /var/log/ceph/cluster1-client.mirror.123279.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128400.log 2024-09-16T13:50:29.349 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.97020.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137818.log.gz 2024-09-16T13:50:29.349 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137860.log.gz 2024-09-16T13:50:29.349 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46011.log 2024-09-16T13:50:29.352 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.128400.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81223.log 2024-09-16T13:50:29.352 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91776.log 2024-09-16T13:50:29.352 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46011.log: /var/log/ceph/cluster1-client.mirror.81223.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73368.log 2024-09-16T13:50:29.352 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91776.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46432.log 2024-09-16T13:50:29.352 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73368.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64627.log 2024-09-16T13:50:29.352 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73368.log.gz 2024-09-16T13:50:29.352 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54905.log 2024-09-16T13:50:29.353 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46432.log: /var/log/ceph/cluster1-client.mirror.64627.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41379.log 2024-09-16T13:50:29.353 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112672.log.gz 2024-09-16T13:50:29.353 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30011.log 2024-09-16T13:50:29.353 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.41379.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52642.log 2024-09-16T13:50:29.353 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30011.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114099.log.gz 2024-09-16T13:50:29.353 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120711.log 2024-09-16T13:50:29.353 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38495.log 2024-09-16T13:50:29.353 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 /var/log/ceph/cluster1-client.mirror.120711.log: --verbose -- /var/log/ceph/cluster1-client.mirror.45589.log 2024-09-16T13:50:29.357 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39506.log 2024-09-16T13:50:29.357 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.38495.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116854.log 2024-09-16T13:50:29.358 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45589.log: /var/log/ceph/cluster1-client.admin.39506.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138954.log 2024-09-16T13:50:29.358 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39506.log.gz 2024-09-16T13:50:29.358 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127004.log 2024-09-16T13:50:29.358 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116854.log: /var/log/ceph/cluster1-client.mirror.138954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91691.log 2024-09-16T13:50:29.358 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95236.log 2024-09-16T13:50:29.358 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127004.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80146.log.gz 2024-09-16T13:50:29.358 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91691.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75245.log 2024-09-16T13:50:29.358 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85971.log 2024-09-16T13:50:29.358 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.95236.log: /var/log/ceph/cluster1-client.mirror.75245.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135187.log 2024-09-16T13:50:29.358 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111975.log 2024-09-16T13:50:29.358 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.85971.log: /var/log/ceph/cluster1-client.mirror.135187.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45127.log 2024-09-16T13:50:29.359 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111975.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113679.log 2024-09-16T13:50:29.359 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91988.log.gz 2024-09-16T13:50:29.359 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45127.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119369.log 2024-09-16T13:50:29.359 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113679.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81558.log 2024-09-16T13:50:29.359 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87152.log 2024-09-16T13:50:29.359 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.119369.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119369.log.gz 2024-09-16T13:50:29.359 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92113.log.gz 2024-09-16T13:50:29.359 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81558.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64977.log 2024-09-16T13:50:29.364 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87152.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123636.log 2024-09-16T13:50:29.365 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38028.log 2024-09-16T13:50:29.365 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64977.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55958.log 2024-09-16T13:50:29.365 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123636.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135338.log 2024-09-16T13:50:29.365 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.38028.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38028.log.gz 2024-09-16T13:50:29.365 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81833.log.gz 2024-09-16T13:50:29.365 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55958.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102096.log 2024-09-16T13:50:29.365 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135338.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136220.log 2024-09-16T13:50:29.365 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102096.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61125.log 2024-09-16T13:50:29.365 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136220.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64236.log 2024-09-16T13:50:29.365 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111614.log 2024-09-16T13:50:29.365 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61125.log: /var/log/ceph/cluster1-client.mirror.64236.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45068.log 2024-09-16T13:50:29.365 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105501.log 2024-09-16T13:50:29.366 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111614.log: /var/log/ceph/cluster1-client.mirror.45068.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144352.log 2024-09-16T13:50:29.366 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88660.log 2024-09-16T13:50:29.366 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105501.log: /var/log/ceph/cluster2-client.admin.144352.log: 90.0% 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140094.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.144352.log.gz 2024-09-16T13:50:29.366 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:29.366 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112207.log 2024-09-16T13:50:29.366 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88660.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117051.log 2024-09-16T13:50:29.366 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112207.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137752.log 2024-09-16T13:50:29.366 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117051.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.117051.log.gz 2024-09-16T13:50:29.366 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99842.log 2024-09-16T13:50:29.366 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97918.log 2024-09-16T13:50:29.366 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74844.log 2024-09-16T13:50:29.370 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.97918.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28848.log 2024-09-16T13:50:29.370 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.74844.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84027.log 2024-09-16T13:50:29.371 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28848.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115653.log 2024-09-16T13:50:29.371 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113047.log 2024-09-16T13:50:29.371 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.84027.log: /var/log/ceph/cluster1-client.mirror.115653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52746.log 2024-09-16T13:50:29.371 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113047.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125316.log.gz 2024-09-16T13:50:29.371 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103902.log 2024-09-16T13:50:29.371 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116580.log 2024-09-16T13:50:29.371 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103902.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125948.log 2024-09-16T13:50:29.371 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116580.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104954.log 2024-09-16T13:50:29.371 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90558.log 2024-09-16T13:50:29.371 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125948.log: /var/log/ceph/cluster1-client.mirror.104954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81667.log 2024-09-16T13:50:29.372 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68679.log 2024-09-16T13:50:29.372 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90558.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29550.log 2024-09-16T13:50:29.372 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81667.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29296.log 2024-09-16T13:50:29.372 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68679.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54781.log 2024-09-16T13:50:29.372 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29550.log: /var/log/ceph/cluster1-client.mirror.29296.log: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.96285.log.gz 2024-09-16T13:50:29.372 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93928.log 2024-09-16T13:50:29.376 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54781.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25942.log 2024-09-16T13:50:29.377 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138634.log 2024-09-16T13:50:29.377 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93928.log: /var/log/ceph/cluster2-client.admin.25942.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25942.log.gz 2024-09-16T13:50:29.377 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61831.log 2024-09-16T13:50:29.377 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138634.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96461.log 2024-09-16T13:50:29.377 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52641.log 2024-09-16T13:50:29.377 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.61831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112520.log 2024-09-16T13:50:29.377 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96461.log: /var/log/ceph/cluster1-client.mirror.52641.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67414.log 2024-09-16T13:50:29.377 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125191.log.gz 2024-09-16T13:50:29.377 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112520.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34213.log 2024-09-16T13:50:29.377 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67414.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104153.log 2024-09-16T13:50:29.377 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34213.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103858.log 2024-09-16T13:50:29.377 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104153.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49131.log 2024-09-16T13:50:29.377 INFO:teuthology.orchestra.run.smithi172.stderr: 18.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.27126.log.gz 2024-09-16T13:50:29.378 INFO:teuthology.orchestra.run.smithi172.stderr: 56.0% -- replaced with /var/log/ceph/cluster2-client.mirror.27108.log.gz 2024-09-16T13:50:29.378 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103858.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33414.log 2024-09-16T13:50:29.378 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114729.log.gz 2024-09-16T13:50:29.378 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136113.log 2024-09-16T13:50:29.378 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33414.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61878.log 2024-09-16T13:50:29.378 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136113.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71270.log 2024-09-16T13:50:29.382 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134538.log 2024-09-16T13:50:29.382 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.61878.log: /var/log/ceph/cluster1-client.admin.71270.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95521.log 2024-09-16T13:50:29.382 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43275.log 2024-09-16T13:50:29.382 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134538.log: /var/log/ceph/cluster2-client.admin.95521.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134749.log 2024-09-16T13:50:29.382 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43275.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54929.log 2024-09-16T13:50:29.382 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134749.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134153.log 2024-09-16T13:50:29.382 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54929.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99706.log.gz 2024-09-16T13:50:29.382 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118947.log 2024-09-16T13:50:29.382 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134153.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106546.log 2024-09-16T13:50:29.382 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster2-client.admin.118947.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77250.log 2024-09-16T13:50:29.382 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132294.log 2024-09-16T13:50:29.382 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106546.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119539.log 2024-09-16T13:50:29.383 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77250.log: /var/log/ceph/cluster1-client.mirror.132294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55980.log 2024-09-16T13:50:29.383 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose /var/log/ceph/cluster1-client.admin.119539.log: -- /var/log/ceph/cluster1-client.mirror.136642.log 2024-09-16T13:50:29.386 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55980.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103693.log 2024-09-16T13:50:29.386 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82195.log 2024-09-16T13:50:29.386 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103693.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84743.log 2024-09-16T13:50:29.386 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.82195.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65478.log 2024-09-16T13:50:29.386 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.82195.log.gz 2024-09-16T13:50:29.386 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.84743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99034.log 2024-09-16T13:50:29.386 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65478.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47900.log 2024-09-16T13:50:29.386 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.99034.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126853.log 2024-09-16T13:50:29.386 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.47900.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130212.log 2024-09-16T13:50:29.386 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126853.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123637.log 2024-09-16T13:50:29.387 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.130212.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30012.log.gz 2024-09-16T13:50:29.388 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122040.log 2024-09-16T13:50:29.388 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123637.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40917.log 2024-09-16T13:50:29.388 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122040.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65817.log 2024-09-16T13:50:29.389 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40917.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.36833.log.gz 2024-09-16T13:50:29.389 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124034.log 2024-09-16T13:50:29.390 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75918.log 2024-09-16T13:50:29.390 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65817.log: /var/log/ceph/cluster1-client.mirror.124034.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67202.log 2024-09-16T13:50:29.390 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75918.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55830.log 2024-09-16T13:50:29.390 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.36836.log.gz 2024-09-16T13:50:29.390 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67202.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30011.log.gz 2024-09-16T13:50:29.391 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120574.log 2024-09-16T13:50:29.391 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55830.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136703.log 2024-09-16T13:50:29.392 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125527.log 2024-09-16T13:50:29.393 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.120574.log: /var/log/ceph/cluster1-client.mirror.136703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80194.log 2024-09-16T13:50:29.393 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39619.log.gz 2024-09-16T13:50:29.393 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125527.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83836.log 2024-09-16T13:50:29.393 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90851.log 2024-09-16T13:50:29.393 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80194.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56336.log 2024-09-16T13:50:29.393 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.83836.log: /var/log/ceph/cluster1-client.mirror.90851.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142499.log 2024-09-16T13:50:29.394 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136979.log 2024-09-16T13:50:29.394 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56336.log: gzip /var/log/ceph/cluster2-client.mirror.142499.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.121189.log 2024-09-16T13:50:29.395 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136979.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43335.log 2024-09-16T13:50:29.395 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122728.log 2024-09-16T13:50:29.395 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.121189.log: /var/log/ceph/cluster1-client.mirror.43335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57874.log 2024-09-16T13:50:29.395 INFO:teuthology.orchestra.run.smithi172.stderr: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.27023.log.gz 2.3% 2024-09-16T13:50:29.395 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.27040.log.gz 2024-09-16T13:50:29.396 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122728.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143542.log 2024-09-16T13:50:29.396 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57874.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123699.log 2024-09-16T13:50:29.396 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120732.log 2024-09-16T13:50:29.398 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.143542.log: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.143542.log.gz 2024-09-16T13:50:29.398 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123699.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71675.log 2024-09-16T13:50:29.398 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.120732.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142008.log 2024-09-16T13:50:29.398 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.120732.log.gz 2024-09-16T13:50:29.398 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71675.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36071.log 2024-09-16T13:50:29.399 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142008.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70332.log 2024-09-16T13:50:29.399 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104388.log 2024-09-16T13:50:29.400 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36071.log: /var/log/ceph/cluster1-client.mirror.70332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57809.log 2024-09-16T13:50:29.401 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75330.log 2024-09-16T13:50:29.401 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104388.log: /var/log/ceph/cluster1-client.mirror.57809.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115403.log 2024-09-16T13:50:29.401 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75330.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115505.log 2024-09-16T13:50:29.401 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137440.log.gz 2024-09-16T13:50:29.401 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115403.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33080.log 2024-09-16T13:50:29.401 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115505.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96279.log 2024-09-16T13:50:29.403 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.33080.log: -5 --verbose -- /var/log/ceph/cluster2-client.admin.72047.log 2024-09-16T13:50:29.403 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96279.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67622.log 2024-09-16T13:50:29.403 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42706.log 2024-09-16T13:50:29.403 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.72047.log: /var/log/ceph/cluster1-client.mirror.67622.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101589.log 2024-09-16T13:50:29.404 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77521.log 2024-09-16T13:50:29.404 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42706.log: /var/log/ceph/cluster1-client.mirror.101589.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84214.log 2024-09-16T13:50:29.404 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77521.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78510.log 2024-09-16T13:50:29.406 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.84214.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46802.log 2024-09-16T13:50:29.406 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115716.log 2024-09-16T13:50:29.406 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.46802.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80825.log.gz 2024-09-16T13:50:29.406 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46802.log.gz 2024-09-16T13:50:29.406 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110250.log 2024-09-16T13:50:29.406 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58228.log 2024-09-16T13:50:29.406 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.110250.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115655.log 2024-09-16T13:50:29.411 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71272.log 2024-09-16T13:50:29.412 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31611.log 2024-09-16T13:50:29.412 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71272.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130802.log 2024-09-16T13:50:29.412 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31611.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37553.log 2024-09-16T13:50:29.412 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.130802.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113553.log 2024-09-16T13:50:29.412 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37553.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87045.log 2024-09-16T13:50:29.412 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113553.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102498.log.gz 2024-09-16T13:50:29.412 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141582.log 2024-09-16T13:50:29.412 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.87045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94258.log 2024-09-16T13:50:29.412 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87045.log.gz 2024-09-16T13:50:29.412 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141582.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137482.log 2024-09-16T13:50:29.412 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.94258.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.94258.log.gz 2024-09-16T13:50:29.413 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88082.log 2024-09-16T13:50:29.413 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28973.log 2024-09-16T13:50:29.413 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.88082.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41337.log 2024-09-16T13:50:29.413 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.88082.log.gz 2024-09-16T13:50:29.413 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28973.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53752.log 2024-09-16T13:50:29.413 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41337.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126582.log 2024-09-16T13:50:29.417 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81811.log 2024-09-16T13:50:29.418 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126582.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76800.log 2024-09-16T13:50:29.418 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81811.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93588.log 2024-09-16T13:50:29.418 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76800.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135063.log 2024-09-16T13:50:29.418 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93461.log 2024-09-16T13:50:29.418 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93588.log: /var/log/ceph/cluster1-client.mirror.135063.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122624.log 2024-09-16T13:50:29.418 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137398.log 2024-09-16T13:50:29.418 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93461.log: /var/log/ceph/cluster1-client.mirror.122624.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137122.log 2024-09-16T13:50:29.418 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89171.log 2024-09-16T13:50:29.418 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30219.log 2024-09-16T13:50:29.418 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89171.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121249.log 2024-09-16T13:50:29.418 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30219.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52899.log 2024-09-16T13:50:29.418 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121249.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84792.log 2024-09-16T13:50:29.418 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52899.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67158.log 2024-09-16T13:50:29.419 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.84792.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79200.log.gz 2024-09-16T13:50:29.435 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133379.log 2024-09-16T13:50:29.435 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130076.log 2024-09-16T13:50:29.435 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67158.log: /var/log/ceph/cluster1-client.admin.133379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136281.log 2024-09-16T13:50:29.435 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130076.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46928.log 2024-09-16T13:50:29.435 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136281.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42642.log 2024-09-16T13:50:29.435 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46928.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38454.log 2024-09-16T13:50:29.435 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42642.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46432.log.gz 2024-09-16T13:50:29.435 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72041.log 2024-09-16T13:50:29.435 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.38454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29842.log 2024-09-16T13:50:29.435 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.72041.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72041.log.gz 2024-09-16T13:50:29.435 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137751.log 2024-09-16T13:50:29.436 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91357.log 2024-09-16T13:50:29.436 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137751.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41379.log.gz 2024-09-16T13:50:29.436 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41376.log.gz 2024-09-16T13:50:29.436 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112879.log 2024-09-16T13:50:29.436 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91357.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45128.log 2024-09-16T13:50:29.436 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110186.log 2024-09-16T13:50:29.436 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112879.log: /var/log/ceph/cluster1-client.mirror.45128.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31269.log 2024-09-16T13:50:29.436 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.48547.log.gz 2024-09-16T13:50:29.436 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110186.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108017.log 2024-09-16T13:50:29.436 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88289.log 2024-09-16T13:50:29.436 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108017.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91608.log 2024-09-16T13:50:29.436 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.108017.log.gz 2024-09-16T13:50:29.436 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85069.log 2024-09-16T13:50:29.437 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.88289.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.49437.log.gz 2024-09-16T13:50:29.437 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91608.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136558.log 2024-09-16T13:50:29.437 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85069.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87667.log 2024-09-16T13:50:29.437 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136558.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84562.log 2024-09-16T13:50:29.437 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119854.log 2024-09-16T13:50:29.437 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87667.log: /var/log/ceph/cluster2-client.admin.84562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64396.log 2024-09-16T13:50:29.437 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130179.log 2024-09-16T13:50:29.437 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.119854.log: /var/log/ceph/cluster1-client.mirror.64396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47253.log 2024-09-16T13:50:29.437 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61562.log 2024-09-16T13:50:29.437 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.130179.log: /var/log/ceph/cluster1-client.admin.47253.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27865.log 2024-09-16T13:50:29.437 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102306.log 2024-09-16T13:50:29.437 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.61562.log: /var/log/ceph/cluster1-client.mirror.27865.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45863.log.gz 2024-09-16T13:50:29.437 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43594.log 2024-09-16T13:50:29.437 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.102306.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.50548.log 2024-09-16T13:50:29.438 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43594.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56189.log 2024-09-16T13:50:29.438 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114979.log 2024-09-16T13:50:29.438 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.50548.log: /var/log/ceph/cluster1-client.mirror.56189.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73424.log 2024-09-16T13:50:29.438 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118945.log 2024-09-16T13:50:29.438 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114979.log: /var/log/ceph/cluster1-client.admin.73424.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123303.log 2024-09-16T13:50:29.438 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103015.log 2024-09-16T13:50:29.438 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.118945.log.gz 2024-09-16T13:50:29.438 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105020.log 2024-09-16T13:50:29.438 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103015.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57092.log 2024-09-16T13:50:29.438 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.105020.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69605.log 2024-09-16T13:50:29.438 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57092.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114771.log 2024-09-16T13:50:29.438 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69605.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112783.log 2024-09-16T13:50:29.438 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30680.log.gz 2024-09-16T13:50:29.438 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69055.log 2024-09-16T13:50:29.439 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112783.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103271.log 2024-09-16T13:50:29.446 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103759.log 2024-09-16T13:50:29.446 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69055.log: /var/log/ceph/cluster1-client.mirror.103271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46009.log 2024-09-16T13:50:29.446 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117644.log 2024-09-16T13:50:29.446 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46009.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71978.log 2024-09-16T13:50:29.446 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79746.log 2024-09-16T13:50:29.446 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.71978.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43652.log 2024-09-16T13:50:29.446 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30889.log 2024-09-16T13:50:29.447 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43652.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79140.log.gz 2024-09-16T13:50:29.447 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.28888.log.gz 2024-09-16T13:50:29.447 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95219.log 2024-09-16T13:50:29.447 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30889.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32452.log 2024-09-16T13:50:29.447 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43381.log 2024-09-16T13:50:29.447 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.95219.log: /var/log/ceph/cluster1-client.mirror.32452.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90181.log 2024-09-16T13:50:29.447 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43381.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111131.log 2024-09-16T13:50:29.447 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90181.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131165.log 2024-09-16T13:50:29.448 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86240.log 2024-09-16T13:50:29.448 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.131165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129901.log 2024-09-16T13:50:29.448 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40833.log 2024-09-16T13:50:29.448 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.86240.log: /var/log/ceph/cluster1-client.admin.129901.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94031.log 2024-09-16T13:50:29.448 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126498.log 2024-09-16T13:50:29.448 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.94031.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56061.log 2024-09-16T13:50:29.448 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40917.log.gz 2024-09-16T13:50:29.448 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126498.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124687.log 2024-09-16T13:50:29.448 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56061.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75195.log 2024-09-16T13:50:29.448 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41913.log 2024-09-16T13:50:29.448 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29463.log 2024-09-16T13:50:29.449 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.75195.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75195.log.gz 2024-09-16T13:50:29.449 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41913.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63143.log 2024-09-16T13:50:29.449 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29463.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57030.log 2024-09-16T13:50:29.449 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36828.log 2024-09-16T13:50:29.449 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.63143.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63143.log.gz 2024-09-16T13:50:29.449 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57030.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67291.log.gz 2024-09-16T13:50:29.456 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42754.log 2024-09-16T13:50:29.456 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.36828.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102429.log 2024-09-16T13:50:29.456 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42754.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44014.log 2024-09-16T13:50:29.456 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102429.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32110.log 2024-09-16T13:50:29.456 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86775.log 2024-09-16T13:50:29.456 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44014.log: /var/log/ceph/cluster1-client.mirror.32110.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79746.log.gz 2024-09-16T13:50:29.456 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51702.log 2024-09-16T13:50:29.456 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.86775.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77142.log 2024-09-16T13:50:29.456 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71503.log 2024-09-16T13:50:29.456 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77142.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81811.log.gz 2024-09-16T13:50:29.457 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30934.log 2024-09-16T13:50:29.457 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.84734.log.gz 2024-09-16T13:50:29.457 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.84743.log.gz 2024-09-16T13:50:29.457 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71503.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92030.log 2024-09-16T13:50:29.457 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30934.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38003.log 2024-09-16T13:50:29.457 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.92030.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31940.log 2024-09-16T13:50:29.457 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.38003.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127340.log 2024-09-16T13:50:29.457 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38003.log.gz 2024-09-16T13:50:29.457 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31940.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112880.log 2024-09-16T13:50:29.457 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127340.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64235.log 2024-09-16T13:50:29.457 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71501.log 2024-09-16T13:50:29.457 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64235.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.84214.log.gz 2024-09-16T13:50:29.457 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75460.log 2024-09-16T13:50:29.458 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71501.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63844.log 2024-09-16T13:50:29.458 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81265.log 2024-09-16T13:50:29.458 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.admin.63844.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123847.log 2024-09-16T13:50:29.458 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63844.log.gz 2024-09-16T13:50:29.458 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.81265.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.73511.log 2024-09-16T13:50:29.458 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123847.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114872.log 2024-09-16T13:50:29.458 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.73511.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58641.log 2024-09-16T13:50:29.463 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114872.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56820.log 2024-09-16T13:50:29.463 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.58641.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.58641.log.gz 2024-09-16T13:50:29.463 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120062.log 2024-09-16T13:50:29.463 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98108.log 2024-09-16T13:50:29.463 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103693.log.gz 2024-09-16T13:50:29.463 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.120062.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50460.log 2024-09-16T13:50:29.464 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98108.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93630.log 2024-09-16T13:50:29.464 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.50460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44324.log 2024-09-16T13:50:29.464 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93630.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79958.log 2024-09-16T13:50:29.464 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44324.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123930.log 2024-09-16T13:50:29.464 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79958.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103858.log.gz 2024-09-16T13:50:29.464 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124896.log 2024-09-16T13:50:29.464 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123930.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107923.log 2024-09-16T13:50:29.464 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124896.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57428.log 2024-09-16T13:50:29.464 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28645.log 2024-09-16T13:50:29.465 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.107923.log: /var/log/ceph/cluster1-client.mirror.57428.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117643.log 2024-09-16T13:50:29.465 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105016.log 2024-09-16T13:50:29.465 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117643.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113469.log 2024-09-16T13:50:29.465 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105016.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66870.log 2024-09-16T13:50:29.465 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113469.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85081.log 2024-09-16T13:50:29.465 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66870.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139645.log 2024-09-16T13:50:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85081.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136282.log 2024-09-16T13:50:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138951.log 2024-09-16T13:50:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136219.log 2024-09-16T13:50:29.470 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136282.log: /var/log/ceph/cluster1-client.mirror.138951.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72354.log 2024-09-16T13:50:29.471 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136219.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-09-16T13:50:29.471 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.72354.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27823.log 2024-09-16T13:50:29.471 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129668.log 2024-09-16T13:50:29.471 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.27823.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107353.log 2024-09-16T13:50:29.471 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129668.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56483.log 2024-09-16T13:50:29.471 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.107353.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59550.log 2024-09-16T13:50:29.471 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56483.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50480.log 2024-09-16T13:50:29.471 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59550.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107354.log 2024-09-16T13:50:29.471 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.50480.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96281.log 2024-09-16T13:50:29.471 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.107354.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108255.log 2024-09-16T13:50:29.472 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96281.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104596.log.gz 2024-09-16T13:50:29.472 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102176.log 2024-09-16T13:50:29.472 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108255.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.108255.log.gz 2024-09-16T13:50:29.472 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58166.log 2024-09-16T13:50:29.472 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102176.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139989.log 2024-09-16T13:50:29.473 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58166.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79514.log.gz 2024-09-16T13:50:29.473 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90014.log 2024-09-16T13:50:29.473 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.139989.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.139989.log.gz 2024-09-16T13:50:29.474 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87201.log 2024-09-16T13:50:29.474 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90014.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56821.log 2024-09-16T13:50:29.474 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.87201.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96927.log 2024-09-16T13:50:29.474 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26382.log 2024-09-16T13:50:29.475 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.96927.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115573.log 2024-09-16T13:50:29.475 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.26382.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.26382.log.gz 2024-09-16T13:50:29.475 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95831.log 2024-09-16T13:50:29.475 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115573.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144433.log 2024-09-16T13:50:29.478 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.95831.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105020.log.gz 2024-09-16T13:50:29.478 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82245.log 2024-09-16T13:50:29.478 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.144433.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87644.log 2024-09-16T13:50:29.478 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144433.log.gz 2024-09-16T13:50:29.478 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.82245.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.82245.log.gz 2024-09-16T13:50:29.478 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80503.log 2024-09-16T13:50:29.478 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33625.log 2024-09-16T13:50:29.478 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80503.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42075.log 2024-09-16T13:50:29.478 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85971.log.gz 2024-09-16T13:50:29.478 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33625.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59548.log 2024-09-16T13:50:29.478 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102805.log 2024-09-16T13:50:29.479 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42075.log: /var/log/ceph/cluster1-client.admin.59548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38589.log 2024-09-16T13:50:29.481 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58737.log 2024-09-16T13:50:29.481 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102805.log: /var/log/ceph/cluster1-client.mirror.38589.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.140883.log 2024-09-16T13:50:29.481 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58737.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33793.log 2024-09-16T13:50:29.481 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.140883.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120223.log 2024-09-16T13:50:29.481 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128118.log 2024-09-16T13:50:29.482 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92536.log.gz 2024-09-16T13:50:29.482 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.120223.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77354.log 2024-09-16T13:50:29.482 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.128118.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121410.log 2024-09-16T13:50:29.482 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77354.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116074.log 2024-09-16T13:50:29.482 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121410.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48679.log 2024-09-16T13:50:29.482 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116074.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39593.log 2024-09-16T13:50:29.484 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.48679.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137607.log 2024-09-16T13:50:29.484 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138534.log 2024-09-16T13:50:29.484 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39593.log: /var/log/ceph/cluster1-client.mirror.137607.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123489.log 2024-09-16T13:50:29.484 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138534.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68128.log 2024-09-16T13:50:29.484 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107174.log 2024-09-16T13:50:29.485 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123489.log: /var/log/ceph/cluster1-client.mirror.68128.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30094.log 2024-09-16T13:50:29.485 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107174.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51087.log.gz 2024-09-16T13:50:29.485 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.75330.log.gz 2024-09-16T13:50:29.485 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49118.log 2024-09-16T13:50:29.487 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30094.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69266.log 2024-09-16T13:50:29.488 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.75918.log.gz 2024-09-16T13:50:29.488 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49118.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92367.log 2024-09-16T13:50:29.488 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49118.log.gz 2024-09-16T13:50:29.488 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69266.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125190.log 2024-09-16T13:50:29.488 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92367.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81918.log 2024-09-16T13:50:29.488 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125190.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94268.log 2024-09-16T13:50:29.488 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91230.log 2024-09-16T13:50:29.488 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81918.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117803.log 2024-09-16T13:50:29.488 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.94268.log: 0.0%/var/log/ceph/cluster1-client.mirror.91230.log: -- replaced with /var/log/ceph/cluster1-client.admin.94268.log.gz 2024-09-16T13:50:29.488 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107575.log 2024-09-16T13:50:29.488 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117803.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88593.log 2024-09-16T13:50:29.488 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107575.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99216.log 2024-09-16T13:50:29.491 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88593.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96273.log 2024-09-16T13:50:29.491 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99216.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70336.log 2024-09-16T13:50:29.491 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93041.log.gz 2024-09-16T13:50:29.492 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96273.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73833.log 2024-09-16T13:50:29.492 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.70336.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51574.log 2024-09-16T13:50:29.492 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75022.log 2024-09-16T13:50:29.492 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.73833.log: /var/log/ceph/cluster1-client.mirror.51574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79983.log 2024-09-16T13:50:29.492 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.admin.75022.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133513.log 2024-09-16T13:50:29.492 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79983.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120595.log 2024-09-16T13:50:29.492 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133513.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81433.log 2024-09-16T13:50:29.492 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.120595.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49837.log 2024-09-16T13:50:29.492 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.120595.log.gz 2024-09-16T13:50:29.496 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142956.log 2024-09-16T13:50:29.496 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.49837.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98633.log 2024-09-16T13:50:29.496 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81433.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133821.log.gz 2024-09-16T13:50:29.496 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142956.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142956.log.gz 2024-09-16T13:50:29.496 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87884.log 2024-09-16T13:50:29.496 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98633.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102090.log 2024-09-16T13:50:29.496 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87884.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102682.log 2024-09-16T13:50:29.496 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102090.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46927.log 2024-09-16T13:50:29.496 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102682.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126011.log 2024-09-16T13:50:29.496 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75063.log 2024-09-16T13:50:29.496 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46927.log: /var/log/ceph/cluster1-client.mirror.126011.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130897.log 2024-09-16T13:50:29.497 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9%/var/log/ceph/cluster1-client.mirror.75063.log: -- replaced with /var/log/ceph/cluster1-client.mirror.115573.log.gzgzip 2024-09-16T13:50:29.497 INFO:teuthology.orchestra.run.smithi172.stderr:-5 --verbose -- /var/log/ceph/cluster1-client.mirror.78254.log 2024-09-16T13:50:29.497 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130897.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111405.log 2024-09-16T13:50:29.497 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78254.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90055.log 2024-09-16T13:50:29.499 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111789.log 2024-09-16T13:50:29.500 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111405.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139669.log.gz 2024-09-16T13:50:29.500 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90055.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121249.log.gz 2024-09-16T13:50:29.500 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32450.log 2024-09-16T13:50:29.500 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111789.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86354.log 2024-09-16T13:50:29.500 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32450.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83619.log 2024-09-16T13:50:29.500 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.86354.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116496.log.gz 2024-09-16T13:50:29.500 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25851.log 2024-09-16T13:50:29.500 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130075.log 2024-09-16T13:50:29.500 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.83619.log: /var/log/ceph/cluster2-client.admin.25851.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89972.log 2024-09-16T13:50:29.500 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25851.log.gz 2024-09-16T13:50:29.500 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130075.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101800.log 2024-09-16T13:50:29.502 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89972.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100604.log 2024-09-16T13:50:29.502 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.101800.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.121592.log 2024-09-16T13:50:29.503 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100604.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.84544.log.gz 2024-09-16T13:50:29.503 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75197.log 2024-09-16T13:50:29.503 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.121592.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87711.log 2024-09-16T13:50:29.503 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121592.log.gz 2024-09-16T13:50:29.503 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.75197.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71506.log 2024-09-16T13:50:29.503 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.87711.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75979.log 2024-09-16T13:50:29.505 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71506.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64975.log 2024-09-16T13:50:29.505 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75979.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31270.log 2024-09-16T13:50:29.505 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64975.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40094.log 2024-09-16T13:50:29.505 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31270.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113679.log.gz 2024-09-16T13:50:29.505 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86488.log 2024-09-16T13:50:29.505 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40094.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74709.log 2024-09-16T13:50:29.505 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.86488.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52019.log 2024-09-16T13:50:29.507 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131232.log 2024-09-16T13:50:29.507 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52019.log: /var/log/ceph/cluster1-client.mirror.74709.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105672.log 2024-09-16T13:50:29.507 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.130210.log.gz 2024-09-16T13:50:29.507 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.131232.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130887.log 2024-09-16T13:50:29.508 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.140883.log.gz 2024-09-16T13:50:29.508 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.105672.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105672.log.gz 2024-09-16T13:50:29.508 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51182.log 2024-09-16T13:50:29.511 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130887.log: 0.0%gzip -- replaced with /var/log/ceph/cluster2-client.admin.130887.log.gz-5 2024-09-16T13:50:29.511 INFO:teuthology.orchestra.run.smithi172.stderr:--verbose -- /var/log/ceph/cluster1-client.mirror.137793.log 2024-09-16T13:50:29.511 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.51182.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130354.log 2024-09-16T13:50:29.511 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61536.log 2024-09-16T13:50:29.512 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130354.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130354.log.gz 2024-09-16T13:50:29.512 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52464.log 2024-09-16T13:50:29.512 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.61536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92704.log 2024-09-16T13:50:29.512 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76198.log 2024-09-16T13:50:29.512 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92704.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67203.log 2024-09-16T13:50:29.512 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112880.log.gz 2024-09-16T13:50:29.512 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76198.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118970.log 2024-09-16T13:50:29.512 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67203.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93085.log 2024-09-16T13:50:29.512 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118970.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103945.log 2024-09-16T13:50:29.518 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93085.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105374.log 2024-09-16T13:50:29.518 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103945.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.137752.log.gz 2024-09-16T13:50:29.518 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137751.log.gz 2024-09-16T13:50:29.518 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53963.log 2024-09-16T13:50:29.518 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100703.log 2024-09-16T13:50:29.518 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42285.log 2024-09-16T13:50:29.518 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53963.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.129307.log.gz 2024-09-16T13:50:29.518 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112628.log 2024-09-16T13:50:29.518 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42285.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48051.log 2024-09-16T13:50:29.518 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112628.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96368.log 2024-09-16T13:50:29.518 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.48051.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144514.log 2024-09-16T13:50:29.518 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136703.log.gz 2024-09-16T13:50:29.519 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96368.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28975.log 2024-09-16T13:50:29.519 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.144514.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144514.log.gz 2024-09-16T13:50:29.519 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93251.log 2024-09-16T13:50:29.519 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136769.log.gz 2024-09-16T13:50:29.519 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28975.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30260.log 2024-09-16T13:50:29.519 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93251.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58059.log 2024-09-16T13:50:29.519 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30260.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43902.log 2024-09-16T13:50:29.519 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58059.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74553.log 2024-09-16T13:50:29.519 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138027.log 2024-09-16T13:50:29.519 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43902.log: gzip -5 /var/log/ceph/cluster1-client.mirror.74553.log: --verbose -- /var/log/ceph/cluster2-client.admin.144595.log 2024-09-16T13:50:29.524 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138027.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132123.log 2024-09-16T13:50:29.524 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126371.log 2024-09-16T13:50:29.524 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.144595.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144595.log.gz 2024-09-16T13:50:29.525 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73397.log 2024-09-16T13:50:29.525 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.132123.log: /var/log/ceph/cluster1-client.mirror.126371.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128243.log 2024-09-16T13:50:29.525 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130992.log 2024-09-16T13:50:29.525 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.73397.log: /var/log/ceph/cluster1-client.admin.128243.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.128243.log.gz 2024-09-16T13:50:29.525 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99542.log 2024-09-16T13:50:29.525 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126853.log.gz 2024-09-16T13:50:29.525 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89003.log 2024-09-16T13:50:29.525 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99542.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100537.log 2024-09-16T13:50:29.525 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67054.log 2024-09-16T13:50:29.525 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89003.log: /var/log/ceph/cluster1-client.mirror.100537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139265.log 2024-09-16T13:50:29.525 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67054.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32993.log 2024-09-16T13:50:29.525 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139265.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37528.log 2024-09-16T13:50:29.525 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105623.log 2024-09-16T13:50:29.526 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121073.log 2024-09-16T13:50:29.526 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37528.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37528.log.gz 2024-09-16T13:50:29.526 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96257.log 2024-09-16T13:50:29.526 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.105623.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105623.log.gz 2024-09-16T13:50:29.526 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76483.log 2024-09-16T13:50:29.526 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96257.log: 0.0%gzip -- replaced with /var/log/ceph/cluster2-client.admin.96257.log.gz 2024-09-16T13:50:29.526 INFO:teuthology.orchestra.run.smithi172.stderr:-5 --verbose -- /var/log/ceph/cluster2-client.mirror.119471.log 2024-09-16T13:50:29.527 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76483.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124687.log.gz 2024-09-16T13:50:29.527 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143065.log 2024-09-16T13:50:29.527 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.119471.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136009.log 2024-09-16T13:50:29.527 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.143065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75980.log 2024-09-16T13:50:29.527 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32781.log 2024-09-16T13:50:29.529 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136009.log: /var/log/ceph/cluster1-client.mirror.75980.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123636.log.gz 2024-09-16T13:50:29.529 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123637.log.gz 2024-09-16T13:50:29.529 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144460.log 2024-09-16T13:50:29.529 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32781.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92451.log 2024-09-16T13:50:29.529 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.144460.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144460.log.gz 2024-09-16T13:50:29.529 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.36828.log.gz 2024-09-16T13:50:29.529 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38434.log 2024-09-16T13:50:29.531 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92451.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72170.log 2024-09-16T13:50:29.531 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134327.log 2024-09-16T13:50:29.531 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.38434.log: /var/log/ceph/cluster1-client.admin.72170.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102073.log 2024-09-16T13:50:29.531 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.49837.log.gz 2024-09-16T13:50:29.531 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134327.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122728.log.gz 2024-09-16T13:50:29.532 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78571.log 2024-09-16T13:50:29.532 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39422.log 2024-09-16T13:50:29.533 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78571.log: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.143065.log.gz 2024-09-16T13:50:29.533 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26867.log 2024-09-16T13:50:29.533 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39422.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102639.log 2024-09-16T13:50:29.533 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.26867.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98504.log 2024-09-16T13:50:29.533 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38247.log 2024-09-16T13:50:29.534 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73198.log 2024-09-16T13:50:29.534 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98504.log: /var/log/ceph/cluster1-client.mirror.38247.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89297.log 2024-09-16T13:50:29.535 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73198.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29296.log.gz 2024-09-16T13:50:29.536 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124814.log 2024-09-16T13:50:29.536 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89297.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74403.log 2024-09-16T13:50:29.536 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124814.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.21191.log 2024-09-16T13:50:29.538 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.74403.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86828.log 2024-09-16T13:50:29.538 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.21191.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119968.log 2024-09-16T13:50:29.538 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.21191.log.gz 2024-09-16T13:50:29.538 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143273.log 2024-09-16T13:50:29.538 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.86828.log: /var/log/ceph/cluster1-client.admin.119968.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39940.log 2024-09-16T13:50:29.538 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129402.log 2024-09-16T13:50:29.539 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.143273.log: /var/log/ceph/cluster1-client.mirror.39940.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.143273.log.gz 2024-09-16T13:50:29.539 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134287.log 2024-09-16T13:50:29.539 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39004.log.gz 2024-09-16T13:50:29.539 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129402.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142179.log 2024-09-16T13:50:29.539 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33713.log.gz 2024-09-16T13:50:29.541 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134287.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122432.log 2024-09-16T13:50:29.541 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142179.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134052.log 2024-09-16T13:50:29.541 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122432.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66865.log 2024-09-16T13:50:29.541 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134052.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76505.log 2024-09-16T13:50:29.541 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66865.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38267.log 2024-09-16T13:50:29.541 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76505.log: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.49131.log.gz 2024-09-16T13:50:29.542 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87463.log 2024-09-16T13:50:29.543 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.38267.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31821.log 2024-09-16T13:50:29.543 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87463.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130713.log 2024-09-16T13:50:29.543 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29102.log 2024-09-16T13:50:29.544 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130713.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118961.log 2024-09-16T13:50:29.544 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30094.log.gz 2024-09-16T13:50:29.544 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29102.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57537.log 2024-09-16T13:50:29.544 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118961.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75567.log 2024-09-16T13:50:29.547 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33370.log 2024-09-16T13:50:29.547 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.75567.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119375.log 2024-09-16T13:50:29.547 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33370.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137693.log 2024-09-16T13:50:29.547 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.119375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32950.log 2024-09-16T13:50:29.547 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119375.log.gz 2024-09-16T13:50:29.547 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51702.log.gz 2024-09-16T13:50:29.547 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137693.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61992.log 2024-09-16T13:50:29.547 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32950.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51528.log 2024-09-16T13:50:29.547 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88084.log 2024-09-16T13:50:29.548 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.61992.log: /var/log/ceph/cluster1-client.mirror.51528.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85877.log 2024-09-16T13:50:29.548 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.88084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112840.log 2024-09-16T13:50:29.551 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.85877.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42012.log.gz 2024-09-16T13:50:29.551 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112352.log 2024-09-16T13:50:29.551 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112840.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111540.log 2024-09-16T13:50:29.551 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112352.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32530.log 2024-09-16T13:50:29.551 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111540.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115192.log 2024-09-16T13:50:29.551 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.32530.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68126.log 2024-09-16T13:50:29.551 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115192.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105673.log 2024-09-16T13:50:29.551 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52464.log.gz 2024-09-16T13:50:29.551 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139839.log 2024-09-16T13:50:29.552 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68126.log: /var/log/ceph/cluster1-client.mirror.105673.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27901.log 2024-09-16T13:50:29.552 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139839.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69520.log 2024-09-16T13:50:29.552 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87905.log 2024-09-16T13:50:29.555 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.27901.log: /var/log/ceph/cluster1-client.mirror.69520.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69310.log 2024-09-16T13:50:29.555 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44955.log.gz 2024-09-16T13:50:29.555 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.87905.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87905.log.gz 2024-09-16T13:50:29.555 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80403.log 2024-09-16T13:50:29.556 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69310.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138448.log 2024-09-16T13:50:29.556 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80403.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45068.log.gz 2024-09-16T13:50:29.556 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55561.log 2024-09-16T13:50:29.556 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138448.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63192.log 2024-09-16T13:50:29.556 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45128.log.gz 2024-09-16T13:50:29.556 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55561.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91819.log 2024-09-16T13:50:29.556 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63192.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111606.log 2024-09-16T13:50:29.556 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91819.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97704.log 2024-09-16T13:50:29.556 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111606.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134813.log 2024-09-16T13:50:29.556 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.97704.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113719.log 2024-09-16T13:50:29.559 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134813.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82322.log 2024-09-16T13:50:29.560 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103901.log 2024-09-16T13:50:29.560 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.82322.log: /var/log/ceph/cluster1-client.mirror.113719.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44014.log.gz 2024-09-16T13:50:29.560 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92960.log 2024-09-16T13:50:29.560 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103901.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79031.log 2024-09-16T13:50:29.560 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92960.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66572.log 2024-09-16T13:50:29.560 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79031.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114814.log 2024-09-16T13:50:29.560 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66572.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69983.log 2024-09-16T13:50:29.560 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32362.log 2024-09-16T13:50:29.560 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69983.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122431.log 2024-09-16T13:50:29.560 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32781.log.gz 2024-09-16T13:50:29.560 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32362.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37525.log 2024-09-16T13:50:29.560 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51150.log 2024-09-16T13:50:29.562 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32869.log.gz 2024-09-16T13:50:29.562 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37525.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37525.log.gz 2024-09-16T13:50:29.562 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95114.log 2024-09-16T13:50:29.562 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51150.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103626.log 2024-09-16T13:50:29.562 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.95114.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74517.log 2024-09-16T13:50:29.562 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108402.log 2024-09-16T13:50:29.563 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.74517.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107525.log 2024-09-16T13:50:29.565 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.108402.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124141.log 2024-09-16T13:50:29.565 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107525.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.107525.log.gz 2024-09-16T13:50:29.565 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118944.log 2024-09-16T13:50:29.565 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65479.log.gz 2024-09-16T13:50:29.565 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65478.log.gz 2024-09-16T13:50:29.565 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124141.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87355.log 2024-09-16T13:50:29.565 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118944.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71655.log 2024-09-16T13:50:29.565 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.118944.log.gz 2024-09-16T13:50:29.565 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.87355.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87355.log.gz 2024-09-16T13:50:29.566 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38951.log 2024-09-16T13:50:29.568 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58166.log.gz 2024-09-16T13:50:29.568 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115442.log 2024-09-16T13:50:29.569 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.38951.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68338.log 2024-09-16T13:50:29.569 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123846.log 2024-09-16T13:50:29.569 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68338.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100392.log 2024-09-16T13:50:29.569 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123846.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64653.log 2024-09-16T13:50:29.569 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100392.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53963.log.gz 2024-09-16T13:50:29.569 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53645.log 2024-09-16T13:50:29.569 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35097.log 2024-09-16T13:50:29.569 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124481.log 2024-09-16T13:50:29.573 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103225.log 2024-09-16T13:50:29.573 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.35097.log: /var/log/ceph/cluster1-client.mirror.124481.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102704.log 2024-09-16T13:50:29.573 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103225.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114810.log 2024-09-16T13:50:29.573 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132703.log 2024-09-16T13:50:29.573 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102704.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143106.log 2024-09-16T13:50:29.573 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114810.log: /var/log/ceph/cluster1-client.mirror.132703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80293.log 2024-09-16T13:50:29.573 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.admin.143106.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127296.log 2024-09-16T13:50:29.574 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.74289.log.gz 2024-09-16T13:50:29.574 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80293.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.143106.log.gz 2024-09-16T13:50:29.574 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79304.log 2024-09-16T13:50:29.574 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127296.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38381.log 2024-09-16T13:50:29.574 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79304.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72562.log 2024-09-16T13:50:29.574 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112042.log 2024-09-16T13:50:29.574 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.38381.log: /var/log/ceph/cluster2-client.admin.72562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129291.log 2024-09-16T13:50:29.577 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112042.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96258.log 2024-09-16T13:50:29.577 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129291.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93841.log 2024-09-16T13:50:29.577 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96258.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26989.log 2024-09-16T13:50:29.578 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76505.log.gz 2024-09-16T13:50:29.578 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96258.log.gz 2024-09-16T13:50:29.578 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.76483.log.gz 2024-09-16T13:50:29.578 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93841.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83616.log 2024-09-16T13:50:29.578 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.26989.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29631.log 2024-09-16T13:50:29.578 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.83616.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78193.log.gz 2024-09-16T13:50:29.578 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94356.log 2024-09-16T13:50:29.578 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29631.log: 65.6% -- replaced with /var/log/ceph/cluster1-client.mirror.131232.log.gz 2024-09-16T13:50:29.578 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104322.log 2024-09-16T13:50:29.578 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78254.log.gz 2024-09-16T13:50:29.578 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.94356.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78884.log 2024-09-16T13:50:29.578 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104322.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.27164.log 2024-09-16T13:50:29.582 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.78884.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.97772.log 2024-09-16T13:50:29.582 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 /var/log/ceph/cluster2-client.mirror.0.27164.log: --verbose -- /var/log/ceph/cluster2-client.admin.61474.log 2024-09-16T13:50:29.582 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.97772.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110794.log 2024-09-16T13:50:29.582 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61474.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80675.log 2024-09-16T13:50:29.582 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.61474.log.gz 2024-09-16T13:50:29.582 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110794.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94286.log 2024-09-16T13:50:29.582 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80675.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30639.log 2024-09-16T13:50:29.582 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.94286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108703.log 2024-09-16T13:50:29.582 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.94286.log.gz 2024-09-16T13:50:29.583 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30639.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99032.log 2024-09-16T13:50:29.583 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.108703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109636.log 2024-09-16T13:50:29.583 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.99032.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93587.log 2024-09-16T13:50:29.583 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.99032.log.gz 2024-09-16T13:50:29.583 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87581.log /var/log/ceph/cluster1-client.admin.109636.log: 2024-09-16T13:50:29.583 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93587.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52920.log 2024-09-16T13:50:29.587 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87581.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120870.log 2024-09-16T13:50:29.588 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52920.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131594.log 2024-09-16T13:50:29.588 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.120870.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81308.log 2024-09-16T13:50:29.588 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.131594.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127885.log 2024-09-16T13:50:29.588 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81308.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33410.log 2024-09-16T13:50:29.588 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127885.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107536.log 2024-09-16T13:50:29.588 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33410.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132030.log 2024-09-16T13:50:29.588 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107536.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43652.log.gz 2024-09-16T13:50:29.588 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137819.log 2024-09-16T13:50:29.588 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116138.log 2024-09-16T13:50:29.588 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138172.log 2024-09-16T13:50:29.588 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124140.log 2024-09-16T13:50:29.588 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138172.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64001.log 2024-09-16T13:50:29.588 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143703.log 2024-09-16T13:50:29.588 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64001.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27006.log 2024-09-16T13:50:29.589 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.143703.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.83616.log.gz 2024-09-16T13:50:29.591 INFO:teuthology.orchestra.run.smithi172.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-client.admin.143703.log.gz 2024-09-16T13:50:29.591 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103968.log 2024-09-16T13:50:29.591 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.27006.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43276.log 2024-09-16T13:50:29.591 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103968.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122668.log 2024-09-16T13:50:29.591 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38723.log 2024-09-16T13:50:29.591 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80293.log.gz/var/log/ceph/cluster1-client.mirror.43276.log: 2024-09-16T13:50:29.592 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122668.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54025.log 2024-09-16T13:50:29.592 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.38723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111830.log 2024-09-16T13:50:29.592 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54025.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143635.log 2024-09-16T13:50:29.592 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111830.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101800.log.gz 2024-09-16T13:50:29.592 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96068.log 2024-09-16T13:50:29.592 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.143635.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56965.log 2024-09-16T13:50:29.593 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.96068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86032.log 2024-09-16T13:50:29.594 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56965.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102704.log.gz 2024-09-16T13:50:29.594 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102682.log.gz 2024-09-16T13:50:29.595 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87541.log 2024-09-16T13:50:29.595 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.86032.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124456.log 2024-09-16T13:50:29.600 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.87541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133105.log 2024-09-16T13:50:29.600 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124456.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39882.log 2024-09-16T13:50:29.600 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39024.log 2024-09-16T13:50:29.600 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128275.log 2024-09-16T13:50:29.600 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93630.log.gz 2024-09-16T13:50:29.600 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.39024.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79094.log 2024-09-16T13:50:29.600 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.128275.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27970.log 2024-09-16T13:50:29.600 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74081.log 2024-09-16T13:50:29.600 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79094.log: /var/log/ceph/cluster1-client.admin.27970.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63855.log 2024-09-16T13:50:29.600 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.74081.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38864.log 2024-09-16T13:50:29.600 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.143635.log.gz 2024-09-16T13:50:29.601 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.63855.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126453.log 2024-09-16T13:50:29.601 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.38864.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92240.log 2024-09-16T13:50:29.601 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126453.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77142.log.gz 2024-09-16T13:50:29.601 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96277.log 2024-09-16T13:50:29.604 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92240.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48551.log 2024-09-16T13:50:29.604 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96277.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135802.log 2024-09-16T13:50:29.604 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.48551.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120917.log 2024-09-16T13:50:29.604 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135802.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55140.log 2024-09-16T13:50:29.604 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107354.log.gz 2024-09-16T13:50:29.605 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.120917.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98208.log 2024-09-16T13:50:29.605 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41595.log 2024-09-16T13:50:29.605 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.97403.log.gz 2024-09-16T13:50:29.605 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.98208.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98208.log.gz 2024-09-16T13:50:29.605 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72169.log 2024-09-16T13:50:29.605 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41595.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137649.log 2024-09-16T13:50:29.605 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.72169.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97839.log 2024-09-16T13:50:29.605 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137649.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31939.log 2024-09-16T13:50:29.607 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.97839.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103125.log 2024-09-16T13:50:29.608 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31939.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.100392.log.gz 2024-09-16T13:50:29.608 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107724.log 2024-09-16T13:50:29.608 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103125.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128139.log 2024-09-16T13:50:29.608 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107724.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116687.log 2024-09-16T13:50:29.608 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.128139.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89801.log 2024-09-16T13:50:29.608 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43482.log 2024-09-16T13:50:29.608 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89801.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86913.log 2024-09-16T13:50:29.608 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112142.log 2024-09-16T13:50:29.613 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57030.log.gz 2024-09-16T13:50:29.613 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.86913.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122021.log 2024-09-16T13:50:29.613 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112142.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127064.log 2024-09-16T13:50:29.614 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55285.log 2024-09-16T13:50:29.614 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84563.log 2024-09-16T13:50:29.614 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127064.log: /var/log/ceph/cluster1-client.mirror.55285.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40069.log 2024-09-16T13:50:29.614 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43800.log 2024-09-16T13:50:29.614 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.84563.log: gzip /var/log/ceph/cluster1-client.mirror.40069.log: -5 --verbose -- /var/log/ceph/cluster1-client.admin.27376.log 2024-09-16T13:50:29.615 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116034.log 2024-09-16T13:50:29.615 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43800.log: /var/log/ceph/cluster1-client.admin.27376.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41511.log 2024-09-16T13:50:29.615 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.88593.log.gz 2024-09-16T13:50:29.615 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116034.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48463.log 2024-09-16T13:50:29.615 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41511.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100810.log 2024-09-16T13:50:29.615 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.48463.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108130.log 2024-09-16T13:50:29.616 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100810.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75154.log 2024-09-16T13:50:29.616 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108130.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.94356.log.gz 2024-09-16T13:50:29.616 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142554.log 2024-09-16T13:50:29.616 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75154.log: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.88084.log.gz 2024-09-16T13:50:29.616 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29800.log 2024-09-16T13:50:29.624 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142554.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83618.log 2024-09-16T13:50:29.624 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29800.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82205.log 2024-09-16T13:50:29.624 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.83618.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96267.log 2024-09-16T13:50:29.624 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135401.log 2024-09-16T13:50:29.625 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.82205.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.82205.log.gz 2024-09-16T13:50:29.625 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96267.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96267.log.gz 2024-09-16T13:50:29.625 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85131.log 2024-09-16T13:50:29.625 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135401.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72039.log 2024-09-16T13:50:29.625 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114243.log 2024-09-16T13:50:29.625 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85131.log: /var/log/ceph/cluster2-client.admin.72039.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108251.log 2024-09-16T13:50:29.625 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.72039.log.gz 2024-09-16T13:50:29.625 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80503.log.gz 2024-09-16T13:50:29.625 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91903.log 2024-09-16T13:50:29.625 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114243.log: /var/log/ceph/cluster2-client.admin.108251.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.108251.log.gz 2024-09-16T13:50:29.625 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68048.log 2024-09-16T13:50:29.625 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91903.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115613.log 2024-09-16T13:50:29.625 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68048.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89174.log.gz 2024-09-16T13:50:29.626 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33079.log 2024-09-16T13:50:29.626 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115613.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74154.log 2024-09-16T13:50:29.626 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33079.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49259.log 2024-09-16T13:50:29.626 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.74154.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73147.log 2024-09-16T13:50:29.626 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112607.log 2024-09-16T13:50:29.626 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49259.log: /var/log/ceph/cluster2-client.admin.73147.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64320.log 2024-09-16T13:50:29.626 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112607.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82322.log.gz 2024-09-16T13:50:29.626 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120734.log 2024-09-16T13:50:29.626 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.64320.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131419.log 2024-09-16T13:50:29.626 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.64320.log.gz 2024-09-16T13:50:29.626 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115234.log 2024-09-16T13:50:29.626 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.120734.log: /var/log/ceph/cluster2-client.mirror.131419.log: 89.9%gzip -5 -- replaced with /var/log/ceph/cluster1-client.mirror.91357.log.gz--verbose -- 2024-09-16T13:50:29.626 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65267.log 2024-09-16T13:50:29.627 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.129291.log.gz 2024-09-16T13:50:29.627 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115234.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60769.log 2024-09-16T13:50:29.627 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65267.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121811.log 2024-09-16T13:50:29.627 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.129402.log.gz 2024-09-16T13:50:29.627 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.60769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120310.log 2024-09-16T13:50:29.627 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121811.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57514.log 2024-09-16T13:50:29.635 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.120310.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54593.log 2024-09-16T13:50:29.635 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57514.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111405.log.gz 2024-09-16T13:50:29.635 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116771.log 2024-09-16T13:50:29.635 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54593.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62132.log 2024-09-16T13:50:29.635 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108771.log 2024-09-16T13:50:29.635 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116771.log: /var/log/ceph/cluster2-client.mirror.62132.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127760.log 2024-09-16T13:50:29.635 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142324.log 2024-09-16T13:50:29.635 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.108771.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.114814.log.gz 2024-09-16T13:50:29.635 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127760.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133992.log 2024-09-16T13:50:29.635 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.142324.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45381.log 2024-09-16T13:50:29.635 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.133992.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133992.log.gz 2024-09-16T13:50:29.636 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137438.log 2024-09-16T13:50:29.636 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45381.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67710.log 2024-09-16T13:50:29.636 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137438.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132191.log 2024-09-16T13:50:29.636 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67710.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100283.log 2024-09-16T13:50:29.636 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.132191.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66004.log 2024-09-16T13:50:29.636 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38978.log 2024-09-16T13:50:29.636 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138448.log.gz 2024-09-16T13:50:29.636 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66004.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87357.log 2024-09-16T13:50:29.636 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.38978.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129754.log 2024-09-16T13:50:29.636 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.87357.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34970.log 2024-09-16T13:50:29.637 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129754.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51222.log 2024-09-16T13:50:29.637 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.34970.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34970.log.gz 2024-09-16T13:50:29.637 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51258.log 2024-09-16T13:50:29.637 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45441.log 2024-09-16T13:50:29.637 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.131419.log.gz 2024-09-16T13:50:29.637 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75576.log 2024-09-16T13:50:29.637 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45441.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51357.log 2024-09-16T13:50:29.637 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.75576.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50065.log 2024-09-16T13:50:29.637 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51357.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108744.log 2024-09-16T13:50:29.637 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.50065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79871.log 2024-09-16T13:50:29.637 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.108744.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92874.log 2024-09-16T13:50:29.641 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79871.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117228.log 2024-09-16T13:50:29.642 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92874.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88743.log 2024-09-16T13:50:29.642 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.117228.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107324.log 2024-09-16T13:50:29.642 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111325.log 2024-09-16T13:50:29.642 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.107324.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40876.log 2024-09-16T13:50:29.642 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111325.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26486.log 2024-09-16T13:50:29.642 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40876.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109759.log 2024-09-16T13:50:29.642 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.26486.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.26486.log.gz 2024-09-16T13:50:29.642 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140195.log.gz 2024-09-16T13:50:29.642 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73066.log 2024-09-16T13:50:29.642 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.109759.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.109759.log.gz 2024-09-16T13:50:29.642 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104574.log 2024-09-16T13:50:29.642 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73066.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73066.log.gz 2024-09-16T13:50:29.642 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92409.log 2024-09-16T13:50:29.643 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48119.log 2024-09-16T13:50:29.643 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92409.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40033.log 2024-09-16T13:50:29.643 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.48119.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59549.log 2024-09-16T13:50:29.643 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127006.log 2024-09-16T13:50:29.645 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59549.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62893.log 2024-09-16T13:50:29.646 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127006.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124141.log.gz 2024-09-16T13:50:29.646 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124140.log.gz 2024-09-16T13:50:29.646 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80673.log 2024-09-16T13:50:29.646 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.62893.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34751.log 2024-09-16T13:50:29.646 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80673.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67284.log 2024-09-16T13:50:29.646 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34751.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72943.log 2024-09-16T13:50:29.646 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67284.log: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.37553.log.gz 2024-09-16T13:50:29.646 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134749.log.gz 2024-09-16T13:50:29.646 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95333.log 2024-09-16T13:50:29.646 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.72943.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123135.log 2024-09-16T13:50:29.646 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.95333.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29631.log.gz 2024-09-16T13:50:29.646 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74040.log 2024-09-16T13:50:29.650 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123135.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126709.log 2024-09-16T13:50:29.650 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.74040.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69185.log 2024-09-16T13:50:29.650 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126709.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108272.log 2024-09-16T13:50:29.650 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68130.log 2024-09-16T13:50:29.650 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51124.log 2024-09-16T13:50:29.650 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.108272.log.gz 2024-09-16T13:50:29.650 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68130.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66783.log 2024-09-16T13:50:29.650 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51124.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62474.log.gz 2024-09-16T13:50:29.650 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134669.log 2024-09-16T13:50:29.651 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66783.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95736.log 2024-09-16T13:50:29.651 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.38723.log.gz 2024-09-16T13:50:29.651 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134669.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69103.log 2024-09-16T13:50:29.651 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.95736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71968.log 2024-09-16T13:50:29.651 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134811.log 2024-09-16T13:50:29.654 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.71968.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118738.log 2024-09-16T13:50:29.655 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134811.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41720.log 2024-09-16T13:50:29.655 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118738.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116834.log 2024-09-16T13:50:29.655 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43383.log 2024-09-16T13:50:29.655 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116834.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122430.log 2024-09-16T13:50:29.655 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103945.log.gz 2024-09-16T13:50:29.655 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39754.log 2024-09-16T13:50:29.655 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103968.log.gz 2024-09-16T13:50:29.655 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122430.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128267.log 2024-09-16T13:50:29.655 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39754.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66152.log 2024-09-16T13:50:29.655 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.128267.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.128267.log.gz 2024-09-16T13:50:29.655 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88785.log 2024-09-16T13:50:29.655 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66152.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90686.log 2024-09-16T13:50:29.655 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88785.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77942.log 2024-09-16T13:50:29.658 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117204.log 2024-09-16T13:50:29.658 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77942.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69310.log.gz 2024-09-16T13:50:29.658 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92747.log 2024-09-16T13:50:29.658 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.117204.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61924.log 2024-09-16T13:50:29.659 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92747.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90767.log 2024-09-16T13:50:29.659 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.61924.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26564.log 2024-09-16T13:50:29.659 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90767.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85051.log 2024-09-16T13:50:29.659 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.26564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77252.log 2024-09-16T13:50:29.659 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.26564.log.gz 2024-09-16T13:50:29.659 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85051.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85051.log.gz 2024-09-16T13:50:29.659 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57957.log 2024-09-16T13:50:29.659 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77252.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33410.log.gz 2024-09-16T13:50:29.663 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33370.log.gz 2024-09-16T13:50:29.663 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142546.log 2024-09-16T13:50:29.663 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57957.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110994.log 2024-09-16T13:50:29.663 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142546.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68553.log 2024-09-16T13:50:29.663 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110994.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108068.log 2024-09-16T13:50:29.663 INFO:teuthology.orchestra.run.smithi172.stderr: 80.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39689.log.gz 2024-09-16T13:50:29.663 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68553.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118931.log 2024-09-16T13:50:29.663 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54444.log 2024-09-16T13:50:29.663 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118931.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.118931.log.gz 2024-09-16T13:50:29.663 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69311.log 2024-09-16T13:50:29.663 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54444.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136641.log 2024-09-16T13:50:29.664 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69311.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78674.log 2024-09-16T13:50:29.664 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136641.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125589.log 2024-09-16T13:50:29.664 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78674.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62843.log 2024-09-16T13:50:29.666 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125589.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52019.log.gz 2024-09-16T13:50:29.666 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126642.log 2024-09-16T13:50:29.666 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.62843.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51925.log 2024-09-16T13:50:29.666 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62843.log.gz 2024-09-16T13:50:29.666 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.126642.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143387.log 2024-09-16T13:50:29.666 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.51925.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97798.log 2024-09-16T13:50:29.666 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51925.log.gz 2024-09-16T13:50:29.666 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.143387.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.143387.log.gz 2024-09-16T13:50:29.666 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41490.log.gz 2024-09-16T13:50:29.666 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88333.log 2024-09-16T13:50:29.668 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.97798.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138660.log 2024-09-16T13:50:29.668 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66030.log 2024-09-16T13:50:29.668 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88333.log: /var/log/ceph/cluster1-client.mirror.138660.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127990.log 2024-09-16T13:50:29.668 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65189.log 2024-09-16T13:50:29.668 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127990.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112983.log 2024-09-16T13:50:29.670 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65189.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43383.log.gz 2024-09-16T13:50:29.670 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141416.log 2024-09-16T13:50:29.670 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112983.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37601.log 2024-09-16T13:50:29.670 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141416.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59768.log 2024-09-16T13:50:29.670 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91271.log 2024-09-16T13:50:29.670 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59768.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32362.log.gz 2024-09-16T13:50:29.672 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43482.log.gz 2024-09-16T13:50:29.672 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42602.log 2024-09-16T13:50:29.672 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66947.log 2024-09-16T13:50:29.672 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52597.log 2024-09-16T13:50:29.672 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66947.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.61562.log.gz 2024-09-16T13:50:29.672 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131621.log 2024-09-16T13:50:29.675 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52597.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91232.log 2024-09-16T13:50:29.675 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.131621.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125654.log 2024-09-16T13:50:29.675 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.48463.log.gz 2024-09-16T13:50:29.675 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91232.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27863.log 2024-09-16T13:50:29.675 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125654.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76885.log 2024-09-16T13:50:29.675 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.27863.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134853.log 2024-09-16T13:50:29.675 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76885.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124561.log 2024-09-16T13:50:29.675 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134853.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84957.log 2024-09-16T13:50:29.675 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124561.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57242.log 2024-09-16T13:50:29.677 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.84957.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.48551.log.gz 2024-09-16T13:50:29.677 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80989.log 2024-09-16T13:50:29.677 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57242.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68129.log 2024-09-16T13:50:29.677 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51124.log.gz 2024-09-16T13:50:29.677 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80989.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33290.log 2024-09-16T13:50:29.678 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68129.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124396.log 2024-09-16T13:50:29.678 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33290.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123090.log 2024-09-16T13:50:29.679 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80298.log 2024-09-16T13:50:29.679 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123090.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28595.log 2024-09-16T13:50:29.679 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.80298.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113511.log 2024-09-16T13:50:29.679 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.28595.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43800.log.gz 2024-09-16T13:50:29.679 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50179.log 2024-09-16T13:50:29.680 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113511.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89087.log 2024-09-16T13:50:29.682 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.50179.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50366.log 2024-09-16T13:50:29.682 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89087.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116373.log 2024-09-16T13:50:29.682 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.50366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39517.log 2024-09-16T13:50:29.682 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69943.log 2024-09-16T13:50:29.682 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116373.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.59768.log.gz 2024-09-16T13:50:29.682 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.39517.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.27135.log 2024-09-16T13:50:29.682 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69943.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34345.log 2024-09-16T13:50:29.682 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.0.27135.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29379.log 2024-09-16T13:50:29.684 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34345.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46222.log 2024-09-16T13:50:29.684 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46805.log 2024-09-16T13:50:29.685 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55140.log.gz 2024-09-16T13:50:29.685 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114707.log 2024-09-16T13:50:29.685 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46805.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74865.log 2024-09-16T13:50:29.685 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114707.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93968.log 2024-09-16T13:50:29.685 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.74865.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74865.log.gz 2024-09-16T13:50:29.685 INFO:teuthology.orchestra.run.smithi172.stderr:gzip 89.6%-5 -- replaced with /var/log/ceph/cluster1-client.mirror.51258.log.gz--verbose 2024-09-16T13:50:29.685 INFO:teuthology.orchestra.run.smithi172.stderr:-- /var/log/ceph/cluster1-client.admin.84029.log 2024-09-16T13:50:29.688 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93968.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29295.log 2024-09-16T13:50:29.688 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.84029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63960.log 2024-09-16T13:50:29.688 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29295.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132486.log 2024-09-16T13:50:29.688 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63960.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70335.log 2024-09-16T13:50:29.688 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.132486.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56798.log 2024-09-16T13:50:29.688 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.70335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59772.log 2024-09-16T13:50:29.688 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56798.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49978.log 2024-09-16T13:50:29.688 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.34751.log.gz 2024-09-16T13:50:29.696 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59772.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132860.log 2024-09-16T13:50:29.696 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.49978.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106316.log 2024-09-16T13:50:29.696 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132860.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130425.log 2024-09-16T13:50:29.697 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106316.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82231.log 2024-09-16T13:50:29.697 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130425.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65398.log 2024-09-16T13:50:29.697 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49134.log 2024-09-16T13:50:29.697 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116288.log 2024-09-16T13:50:29.697 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.82231.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.82231.log.gz 2024-09-16T13:50:29.697 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56965.log.gz 2024-09-16T13:50:29.697 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49134.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66866.log 2024-09-16T13:50:29.697 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116288.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138911.log 2024-09-16T13:50:29.697 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66866.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61045.log 2024-09-16T13:50:29.697 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138911.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107049.log 2024-09-16T13:50:29.697 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120785.log 2024-09-16T13:50:29.697 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.61045.log.gz 2024-09-16T13:50:29.697 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107049.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102916.log 2024-09-16T13:50:29.698 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.120785.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42962.log.gz 2024-09-16T13:50:29.698 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53669.log 2024-09-16T13:50:29.698 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49137.log 2024-09-16T13:50:29.698 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53669.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93503.log 2024-09-16T13:50:29.698 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59723.log 2024-09-16T13:50:29.698 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49137.log: /var/log/ceph/cluster1-client.mirror.93503.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31522.log 2024-09-16T13:50:29.698 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98370.log 2024-09-16T13:50:29.698 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31522.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88257.log 2024-09-16T13:50:29.698 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.98370.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98370.log.gz 2024-09-16T13:50:29.698 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67284.log.gz 2024-09-16T13:50:29.698 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77204.log 2024-09-16T13:50:29.699 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88257.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77141.log 2024-09-16T13:50:29.699 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77204.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34045.log 2024-09-16T13:50:29.718 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77141.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124057.log 2024-09-16T13:50:29.718 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34045.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131125.log 2024-09-16T13:50:29.718 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52878.log 2024-09-16T13:50:29.718 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124057.log: /var/log/ceph/cluster2-client.admin.131125.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126163.log 2024-09-16T13:50:29.718 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131125.log.gz 2024-09-16T13:50:29.718 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68048.log.gz 2024-09-16T13:50:29.718 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52878.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108952.log 2024-09-16T13:50:29.718 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126163.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138491.log 2024-09-16T13:50:29.718 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.108952.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107945.log 2024-09-16T13:50:29.719 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138491.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66947.log.gz 2024-09-16T13:50:29.719 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42495.log 2024-09-16T13:50:29.719 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97430.log 2024-09-16T13:50:29.719 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42495.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75939.log 2024-09-16T13:50:29.719 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.97430.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60477.log 2024-09-16T13:50:29.719 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.75939.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84559.log 2024-09-16T13:50:29.719 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75939.log.gz 2024-09-16T13:50:29.719 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.60477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134919.log 2024-09-16T13:50:29.719 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.84559.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114730.log 2024-09-16T13:50:29.719 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65267.log.gz 2024-09-16T13:50:29.719 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65189.log.gz 2024-09-16T13:50:29.719 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53249.log 2024-09-16T13:50:29.720 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114730.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86085.log 2024-09-16T13:50:29.720 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53249.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128444.log 2024-09-16T13:50:29.720 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.86085.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54485.log 2024-09-16T13:50:29.720 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.128444.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66030.log.gz 2024-09-16T13:50:29.720 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134666.log 2024-09-16T13:50:29.720 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54485.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61132.log 2024-09-16T13:50:29.720 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134666.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42432.log 2024-09-16T13:50:29.720 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61132.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72636.log 2024-09-16T13:50:29.720 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42432.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124900.log 2024-09-16T13:50:29.720 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.72636.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65943.log 2024-09-16T13:50:29.720 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124900.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31521.log 2024-09-16T13:50:29.720 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65943.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48306.log 2024-09-16T13:50:29.720 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31521.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116457.log 2024-09-16T13:50:29.721 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.48306.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118458.log 2024-09-16T13:50:29.721 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116457.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119388.log 2024-09-16T13:50:29.721 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118458.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64713.log 2024-09-16T13:50:29.721 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.119388.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37530.log 2024-09-16T13:50:29.721 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77252.log.gz 2024-09-16T13:50:29.721 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119388.log.gz 2024-09-16T13:50:29.721 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64713.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26538.log 2024-09-16T13:50:29.721 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37530.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37530.log.gz 2024-09-16T13:50:29.721 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95769.log 2024-09-16T13:50:29.721 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.26538.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.26538.log.gz 2024-09-16T13:50:29.721 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68550.log 2024-09-16T13:50:29.721 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104574.log.gz 2024-09-16T13:50:29.721 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.107049.log.gz 2024-09-16T13:50:29.721 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.95769.log: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.107532.log.gz 2024-09-16T13:50:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134642.log 2024-09-16T13:50:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68550.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56271.log 2024-09-16T13:50:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62401.log 2024-09-16T13:50:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79621.log 2024-09-16T13:50:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.62401.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130422.log 2024-09-16T13:50:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:gzip /var/log/ceph/cluster1-client.mirror.79621.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115044.log 2024-09-16T13:50:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130422.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.38267.log.gz 2024-09-16T13:50:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105680.log 2024-09-16T13:50:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115044.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118336.log 2024-09-16T13:50:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105680.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119176.log 2024-09-16T13:50:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118336.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138068.log 2024-09-16T13:50:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103335.log 2024-09-16T13:50:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.119176.log: /var/log/ceph/cluster1-client.mirror.138068.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37047.log 2024-09-16T13:50:29.722 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138700.log 2024-09-16T13:50:29.723 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37047.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97269.log 2024-09-16T13:50:29.723 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138700.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65897.log 2024-09-16T13:50:29.723 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.97269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57245.log 2024-09-16T13:50:29.723 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65897.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49557.log 2024-09-16T13:50:29.723 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57245.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127885.log.gz 2024-09-16T13:50:29.723 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111113.log 2024-09-16T13:50:29.730 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49557.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80887.log 2024-09-16T13:50:29.730 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111113.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65946.log 2024-09-16T13:50:29.730 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80887.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59985.log 2024-09-16T13:50:29.730 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65946.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130891.log 2024-09-16T13:50:29.730 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93042.log 2024-09-16T13:50:29.731 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130891.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130891.log.gz 2024-09-16T13:50:29.731 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46431.log 2024-09-16T13:50:29.731 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93042.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74683.log 2024-09-16T13:50:29.731 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38931.log 2024-09-16T13:50:29.731 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.74683.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108268.log 2024-09-16T13:50:29.731 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.38931.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132564.log 2024-09-16T13:50:29.731 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108268.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43863.log 2024-09-16T13:50:29.731 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.108268.log.gz 2024-09-16T13:50:29.731 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.97798.log.gz 2024-09-16T13:50:29.731 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132564.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131647.log 2024-09-16T13:50:29.731 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43863.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131875.log 2024-09-16T13:50:29.731 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.131647.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113552.log 2024-09-16T13:50:29.731 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.131875.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102496.log 2024-09-16T13:50:29.731 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113552.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93503.log.gz 2024-09-16T13:50:29.732 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62979.log 2024-09-16T13:50:29.732 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102496.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119585.log 2024-09-16T13:50:29.732 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.62979.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137332.log 2024-09-16T13:50:29.732 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.119585.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45548.log 2024-09-16T13:50:29.732 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137542.log 2024-09-16T13:50:29.732 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104786.log 2024-09-16T13:50:29.732 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137542.log: 89.9%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.75979.log.gz 2024-09-16T13:50:29.732 INFO:teuthology.orchestra.run.smithi172.stderr:-5 --verbose -- /var/log/ceph/cluster1-client.mirror.134455.log 2024-09-16T13:50:29.732 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75980.log.gz 2024-09-16T13:50:29.732 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104786.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60757.log 2024-09-16T13:50:29.732 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134455.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82201.log 2024-09-16T13:50:29.732 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.60757.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118933.log 2024-09-16T13:50:29.732 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.82201.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.82201.log.gz 2024-09-16T13:50:29.738 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.75154.log.gz 2024-09-16T13:50:29.738 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109761.log 2024-09-16T13:50:29.738 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90686.log.gz 2024-09-16T13:50:29.738 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118933.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.118933.log.gz 2024-09-16T13:50:29.738 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135550.log 2024-09-16T13:50:29.738 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.109761.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57241.log 2024-09-16T13:50:29.738 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135550.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127552.log 2024-09-16T13:50:29.738 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57241.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69372.log 2024-09-16T13:50:29.739 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127552.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37532.log 2024-09-16T13:50:29.739 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69372.log: 89.5% 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107324.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.107313.log.gz 2024-09-16T13:50:29.739 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:29.739 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49951.log 2024-09-16T13:50:29.739 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111831.log 2024-09-16T13:50:29.739 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38020.log 2024-09-16T13:50:29.739 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111831.log: /var/log/ceph/cluster1-client.admin.49951.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77983.log.gz 2024-09-16T13:50:29.739 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111215.log 2024-09-16T13:50:29.739 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.38020.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38020.log.gz 2024-09-16T13:50:29.739 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45278.log 2024-09-16T13:50:29.739 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.108952.log.gz 2024-09-16T13:50:29.739 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110287.log 2024-09-16T13:50:29.739 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45278.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126347.log 2024-09-16T13:50:29.740 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110287.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41914.log 2024-09-16T13:50:29.740 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126347.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66366.log 2024-09-16T13:50:29.740 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41914.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109367.log 2024-09-16T13:50:29.740 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31822.log 2024-09-16T13:50:29.749 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.109367.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120606.log 2024-09-16T13:50:29.749 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31822.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93714.log 2024-09-16T13:50:29.749 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.88257.log.gz 2024-09-16T13:50:29.749 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.120606.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118488.log 2024-09-16T13:50:29.749 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93714.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27979.log 2024-09-16T13:50:29.750 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.118488.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89298.log 2024-09-16T13:50:29.750 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.27979.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54382.log 2024-09-16T13:50:29.750 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89298.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63256.log 2024-09-16T13:50:29.750 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122131.log 2024-09-16T13:50:29.750 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63256.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98381.log 2024-09-16T13:50:29.750 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84797.log 2024-09-16T13:50:29.750 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114097.log.gz 2024-09-16T13:50:29.750 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.98381.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128379.log 2024-09-16T13:50:29.750 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.84797.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68633.log 2024-09-16T13:50:29.750 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.128379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34553.log 2024-09-16T13:50:29.750 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68633.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105098.log 2024-09-16T13:50:29.750 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34553.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41656.log 2024-09-16T13:50:29.750 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105098.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119927.log 2024-09-16T13:50:29.750 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41656.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53396.log 2024-09-16T13:50:29.751 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92451.log.gz 2024-09-16T13:50:29.751 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.119927.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54802.log 2024-09-16T13:50:29.751 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53751.log 2024-09-16T13:50:29.751 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54802.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86059.log 2024-09-16T13:50:29.751 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55981.log 2024-09-16T13:50:29.751 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.86059.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130893.log 2024-09-16T13:50:29.751 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55981.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134963.log 2024-09-16T13:50:29.751 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130893.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.120310.log.gz 2024-09-16T13:50:29.751 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65523.log 2024-09-16T13:50:29.751 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80192.log 2024-09-16T13:50:29.751 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65523.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117807.log 2024-09-16T13:50:29.751 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80192.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51665.log 2024-09-16T13:50:29.756 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117807.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119606.log 2024-09-16T13:50:29.757 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51665.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38294.log 2024-09-16T13:50:29.757 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.119606.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58463.log 2024-09-16T13:50:29.757 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.38294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54091.log 2024-09-16T13:50:29.757 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93968.log.gz 2024-09-16T13:50:29.757 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58463.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119834.log 2024-09-16T13:50:29.757 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81411.log 2024-09-16T13:50:29.757 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.119834.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.108744.log.gz 2024-09-16T13:50:29.757 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62360.log 2024-09-16T13:50:29.757 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81411.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121505.log 2024-09-16T13:50:29.757 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.62360.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105651.log 2024-09-16T13:50:29.757 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121505.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104112.log 2024-09-16T13:50:29.757 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40113.log 2024-09-16T13:50:29.757 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104112.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136430.log 2024-09-16T13:50:29.757 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40113.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117642.log 2024-09-16T13:50:29.758 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136430.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92029.log 2024-09-16T13:50:29.758 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117642.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73390.log 2024-09-16T13:50:29.758 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46390.log 2024-09-16T13:50:29.758 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127006.log.gz 2024-09-16T13:50:29.758 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73390.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73390.log.gz 2024-09-16T13:50:29.761 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51617.log 2024-09-16T13:50:29.761 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85573.log 2024-09-16T13:50:29.761 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.51617.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115715.log 2024-09-16T13:50:29.761 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.85573.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94008.log 2024-09-16T13:50:29.761 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93335.log 2024-09-16T13:50:29.761 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83839.log 2024-09-16T13:50:29.761 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93335.log: /var/log/ceph/cluster1-client.mirror.94008.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116771.log.gz 2024-09-16T13:50:29.761 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87097.log 2024-09-16T13:50:29.761 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.83839.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103204.log 2024-09-16T13:50:29.761 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116834.log.gz 2024-09-16T13:50:29.764 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87097.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50843.log 2024-09-16T13:50:29.764 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103204.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130306.log 2024-09-16T13:50:29.764 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.50843.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100956.log 2024-09-16T13:50:29.764 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130306.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119379.log 2024-09-16T13:50:29.764 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100956.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87617.log 2024-09-16T13:50:29.764 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116373.log.gz 2024-09-16T13:50:29.764 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.119379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70299.log 2024-09-16T13:50:29.764 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119379.log.gz 2024-09-16T13:50:29.764 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87617.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76925.log 2024-09-16T13:50:29.766 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.70299.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.70299.log.gz 2024-09-16T13:50:29.766 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116457.log.gz 2024-09-16T13:50:29.766 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81138.log 2024-09-16T13:50:29.766 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76925.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138113.log 2024-09-16T13:50:29.767 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76944.log 2024-09-16T13:50:29.767 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138113.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118941.log 2024-09-16T13:50:29.767 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76944.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125741.log 2024-09-16T13:50:29.769 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92200.log 2024-09-16T13:50:29.769 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.118941.log.gz 2024-09-16T13:50:29.769 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74423.log 2024-09-16T13:50:29.769 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.131875.log.gz 2024-09-16T13:50:29.769 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92200.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116456.log 2024-09-16T13:50:29.769 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.74423.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47875.log 2024-09-16T13:50:29.769 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116456.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55495.log 2024-09-16T13:50:29.769 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.47875.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30303.log 2024-09-16T13:50:29.771 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55495.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124184.log 2024-09-16T13:50:29.771 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121447.log 2024-09-16T13:50:29.771 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123090.log.gz 2024-09-16T13:50:29.771 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124184.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73470.log 2024-09-16T13:50:29.772 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121447.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79560.log 2024-09-16T13:50:29.772 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.73470.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131325.log 2024-09-16T13:50:29.773 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79560.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66254.log 2024-09-16T13:50:29.773 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127552.log.gz 2024-09-16T13:50:29.773 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132926.log 2024-09-16T13:50:29.773 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.131325.log: /var/log/ceph/cluster1-client.mirror.66254.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133603.log 2024-09-16T13:50:29.774 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132926.log: 18.5% -- replaced with /var/log/ceph/cluster1-client.admin.120734.log.gz 2024-09-16T13:50:29.774 INFO:teuthology.orchestra.run.smithi172.stderr: 79.9% 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.27901.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.120785.log.gz 2024-09-16T13:50:29.774 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:29.774 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87178.log 2024-09-16T13:50:29.775 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133603.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73605.log 2024-09-16T13:50:29.775 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87178.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50802.log 2024-09-16T13:50:29.776 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.73605.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87977.log 2024-09-16T13:50:29.776 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.50802.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92158.log 2024-09-16T13:50:29.776 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134537.log.gz 2024-09-16T13:50:29.776 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87977.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141940.log 2024-09-16T13:50:29.776 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92158.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99185.log 2024-09-16T13:50:29.777 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.141940.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134669.log.gz 2024-09-16T13:50:29.778 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134666.log.gz 2024-09-16T13:50:29.778 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134642.log.gz 2024-09-16T13:50:29.778 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76614.log 2024-09-16T13:50:29.778 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127610.log 2024-09-16T13:50:29.779 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76614.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66319.log 2024-09-16T13:50:29.779 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127610.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125589.log.gz 2024-09-16T13:50:29.779 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42120.log 2024-09-16T13:50:29.780 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66319.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120424.log 2024-09-16T13:50:29.780 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42120.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118196.log 2024-09-16T13:50:29.781 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.130893.log.gz 2024-09-16T13:50:29.781 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.120424.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43125.log 2024-09-16T13:50:29.781 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118196.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123220.log 2024-09-16T13:50:29.782 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43125.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132741.log 2024-09-16T13:50:29.782 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123220.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32659.log 2024-09-16T13:50:29.782 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135759.log 2024-09-16T13:50:29.783 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32659.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111257.log 2024-09-16T13:50:29.783 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66424.log 2024-09-16T13:50:29.783 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111257.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121705.log 2024-09-16T13:50:29.784 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66424.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87333.log 2024-09-16T13:50:29.785 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121705.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55012.log 2024-09-16T13:50:29.785 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64607.log 2024-09-16T13:50:29.785 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143656.log 2024-09-16T13:50:29.786 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64607.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29634.log 2024-09-16T13:50:29.786 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.143656.log: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.143656.log.gz 2024-09-16T13:50:29.786 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44971.log 2024-09-16T13:50:29.787 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29634.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142328.log 2024-09-16T13:50:29.788 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44971.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100702.log 2024-09-16T13:50:29.788 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34045.log.gz 2024-09-16T13:50:29.788 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.142328.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62930.log 2024-09-16T13:50:29.788 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115043.log 2024-09-16T13:50:29.789 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.62930.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121033.log 2024-09-16T13:50:29.789 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115043.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86715.log 2024-09-16T13:50:29.789 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.121033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103545.log 2024-09-16T13:50:29.790 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.86715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86732.log 2024-09-16T13:50:29.791 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39372.log 2024-09-16T13:50:29.791 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31822.log.gz 2024-09-16T13:50:29.791 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.86732.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131688.log 2024-09-16T13:50:29.793 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.39372.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39372.log.gz 2024-09-16T13:50:29.793 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85539.log 2024-09-16T13:50:29.793 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.131688.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102175.log 2024-09-16T13:50:29.793 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85539.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85539.log.gz 2024-09-16T13:50:29.793 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81856.log 2024-09-16T13:50:29.794 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102175.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132454.log 2024-09-16T13:50:29.794 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81856.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117033.log 2024-09-16T13:50:29.794 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132454.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39940.log.gz 2024-09-16T13:50:29.794 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39882.log.gz 2024-09-16T13:50:29.795 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39485.log 2024-09-16T13:50:29.797 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.117033.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118841.log 2024-09-16T13:50:29.797 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39485.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100603.log 2024-09-16T13:50:29.797 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.118841.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52256.log 2024-09-16T13:50:29.797 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100603.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55495.log.gz 2024-09-16T13:50:29.797 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124056.log 2024-09-16T13:50:29.797 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.52256.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.52256.log.gz 2024-09-16T13:50:29.797 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32572.log 2024-09-16T13:50:29.811 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124056.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77773.log 2024-09-16T13:50:29.811 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32572.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91523.log 2024-09-16T13:50:29.812 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55620.log.gz 2024-09-16T13:50:29.812 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77773.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104743.log 2024-09-16T13:50:29.812 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91523.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81200.log 2024-09-16T13:50:29.812 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104743.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142950.log 2024-09-16T13:50:29.812 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81200.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113108.log 2024-09-16T13:50:29.812 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142950.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142950.log.gz 2024-09-16T13:50:29.812 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40359.log 2024-09-16T13:50:29.812 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113108.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124395.log 2024-09-16T13:50:29.812 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40359.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51440.log 2024-09-16T13:50:29.812 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124395.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72311.log 2024-09-16T13:50:29.812 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45548.log.gz 2024-09-16T13:50:29.812 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.51440.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51440.log.gz 2024-09-16T13:50:29.812 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63230.log 2024-09-16T13:50:29.813 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.72311.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77310.log 2024-09-16T13:50:29.813 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63230.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58382.log 2024-09-16T13:50:29.813 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77310.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57242.log.gz 2024-09-16T13:50:29.813 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57245.log.gz 2024-09-16T13:50:29.813 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89423.log 2024-09-16T13:50:29.813 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57241.log.gz 2024-09-16T13:50:29.813 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58382.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118311.log 2024-09-16T13:50:29.813 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89423.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84855.log 2024-09-16T13:50:29.813 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118311.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54065.log 2024-09-16T13:50:29.813 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.84855.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78611.log 2024-09-16T13:50:29.813 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103270.log 2024-09-16T13:50:29.813 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78611.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32239.log 2024-09-16T13:50:29.813 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29379.log.gz 2024-09-16T13:50:29.814 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75308.log 2024-09-16T13:50:29.814 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32239.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51355.log 2024-09-16T13:50:29.814 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75308.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56190.log 2024-09-16T13:50:29.814 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51355.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60529.log 2024-09-16T13:50:29.814 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56190.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82025.log 2024-09-16T13:50:29.814 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.60529.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131211.log 2024-09-16T13:50:29.814 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.82025.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60757.log.gz 2024-09-16T13:50:29.814 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125655.log 2024-09-16T13:50:29.814 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.131211.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57115.log 2024-09-16T13:50:29.814 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108838.log 2024-09-16T13:50:29.814 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57115.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41438.log 2024-09-16T13:50:29.814 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.108838.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64607.log.gz 2024-09-16T13:50:29.814 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75871.log 2024-09-16T13:50:29.815 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41438.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34554.log 2024-09-16T13:50:29.815 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75871.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30259.log 2024-09-16T13:50:29.824 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34554.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102596.log 2024-09-16T13:50:29.824 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132475.log 2024-09-16T13:50:29.824 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102596.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142954.log 2024-09-16T13:50:29.824 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.36677.log.gz 2024-09-16T13:50:29.824 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.132475.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108119.log 2024-09-16T13:50:29.825 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.132475.log.gz 2024-09-16T13:50:29.825 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142954.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142954.log.gz 2024-09-16T13:50:29.825 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36831.log 2024-09-16T13:50:29.825 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108119.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118198.log 2024-09-16T13:50:29.825 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.36831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46643.log 2024-09-16T13:50:29.825 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56271.log.gz 2024-09-16T13:50:29.825 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118198.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108630.log 2024-09-16T13:50:29.825 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46643.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31310.log 2024-09-16T13:50:29.825 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.108630.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130478.log 2024-09-16T13:50:29.825 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31310.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105784.log 2024-09-16T13:50:29.825 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130478.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49132.log 2024-09-16T13:50:29.825 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105784.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87488.log 2024-09-16T13:50:29.826 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49132.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66782.log 2024-09-16T13:50:29.826 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87488.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59986.log 2024-09-16T13:50:29.826 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117011.log 2024-09-16T13:50:29.826 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59986.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142551.log 2024-09-16T13:50:29.826 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117011.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.117011.log.gz 2024-09-16T13:50:29.826 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121550.log 2024-09-16T13:50:29.826 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142551.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91736.log 2024-09-16T13:50:29.826 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65946.log.gz 2024-09-16T13:50:29.826 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121550.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102705.log 2024-09-16T13:50:29.826 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91736.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43592.log 2024-09-16T13:50:29.826 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102705.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40877.log 2024-09-16T13:50:29.826 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43592.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143977.log 2024-09-16T13:50:29.826 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103122.log 2024-09-16T13:50:29.827 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.143977.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86962.log 2024-09-16T13:50:29.827 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89843.log 2024-09-16T13:50:29.827 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.86962.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139080.log 2024-09-16T13:50:29.827 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89843.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65164.log 2024-09-16T13:50:29.827 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139080.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38225.log 2024-09-16T13:50:29.831 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65164.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73967.log 2024-09-16T13:50:29.831 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.38225.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74917.log 2024-09-16T13:50:29.832 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.73967.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78316.log 2024-09-16T13:50:29.832 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.74917.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131822.log 2024-09-16T13:50:29.832 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78316.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101877.log 2024-09-16T13:50:29.832 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.131822.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137272.log 2024-09-16T13:50:29.832 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101877.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58082.log 2024-09-16T13:50:29.832 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137187.log 2024-09-16T13:50:29.832 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58082.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58635.log 2024-09-16T13:50:29.832 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.143977.log.gz 2024-09-16T13:50:29.832 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137187.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115654.log 2024-09-16T13:50:29.832 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.58635.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.58635.log.gz 2024-09-16T13:50:29.832 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58335.log 2024-09-16T13:50:29.832 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115654.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76944.log.gz 2024-09-16T13:50:29.833 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.76925.log.gz 2024-09-16T13:50:29.833 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71818.log 2024-09-16T13:50:29.833 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58335.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79560.log.gz 2024-09-16T13:50:29.833 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51307.log 2024-09-16T13:50:29.833 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79621.log.gz 2024-09-16T13:50:29.833 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.71818.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56402.log 2024-09-16T13:50:29.837 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.51307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65312.log 2024-09-16T13:50:29.837 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56402.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53030.log 2024-09-16T13:50:29.837 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26833.log 2024-09-16T13:50:29.837 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33504.log 2024-09-16T13:50:29.837 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.26833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76715.log 2024-09-16T13:50:29.837 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33504.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76614.log.gz 2024-09-16T13:50:29.837 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30769.log 2024-09-16T13:50:29.837 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31060.log 2024-09-16T13:50:29.837 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30769.log: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.97839.log.gz 2024-09-16T13:50:29.837 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51978.log 2024-09-16T13:50:29.837 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31060.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34343.log 2024-09-16T13:50:29.838 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51978.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51034.log 2024-09-16T13:50:29.838 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34343.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104744.log 2024-09-16T13:50:29.838 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.51034.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51034.log.gz 2024-09-16T13:50:29.838 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93927.log 2024-09-16T13:50:29.840 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104744.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86890.log 2024-09-16T13:50:29.840 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93927.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101445.log 2024-09-16T13:50:29.840 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.86890.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96684.log 2024-09-16T13:50:29.840 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101445.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99324.log 2024-09-16T13:50:29.840 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96684.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96684.log.gz 2024-09-16T13:50:29.840 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.97269.log.gz 2024-09-16T13:50:29.840 INFO:teuthology.orchestra.run.smithi172.stderr:gzip 89.9%-5 --verbose -- replaced with /var/log/ceph/cluster1-client.mirror.102496.log.gz-- 2024-09-16T13:50:29.840 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40321.log 2024-09-16T13:50:29.840 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.99324.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.99324.log.gz 2024-09-16T13:50:29.842 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99335.log 2024-09-16T13:50:29.842 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40321.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32739.log 2024-09-16T13:50:29.842 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.99335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139374.log 2024-09-16T13:50:29.842 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119492.log 2024-09-16T13:50:29.844 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121838.log 2024-09-16T13:50:29.844 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.119492.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55916.log 2024-09-16T13:50:29.844 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77310.log.gz 2024-09-16T13:50:29.844 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133002.log 2024-09-16T13:50:29.844 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49806.log 2024-09-16T13:50:29.845 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105120.log 2024-09-16T13:50:29.845 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49806.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49806.log.gz 2024-09-16T13:50:29.845 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55200.log 2024-09-16T13:50:29.845 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105120.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62515.log 2024-09-16T13:50:29.846 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55200.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28331.log 2024-09-16T13:50:29.847 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68550.log.gz 2024-09-16T13:50:29.847 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.62515.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101842.log 2024-09-16T13:50:29.847 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39045.log 2024-09-16T13:50:29.848 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98857.log 2024-09-16T13:50:29.848 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39045.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85075.log 2024-09-16T13:50:29.848 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98857.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95304.log 2024-09-16T13:50:29.848 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85075.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31059.log 2024-09-16T13:50:29.849 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.95304.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79032.log 2024-09-16T13:50:29.850 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31059.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29225.log 2024-09-16T13:50:29.850 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79032.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61068.log 2024-09-16T13:50:29.850 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29225.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80402.log 2024-09-16T13:50:29.851 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61068.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.105784.log.gz 2024-09-16T13:50:29.851 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78782.log 2024-09-16T13:50:29.852 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80402.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80168.log 2024-09-16T13:50:29.852 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143589.log 2024-09-16T13:50:29.852 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72943.log.gz 2024-09-16T13:50:29.854 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32451.log 2024-09-16T13:50:29.854 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.143589.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86199.log 2024-09-16T13:50:29.854 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92029.log.gz 2024-09-16T13:50:29.854 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32451.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43653.log 2024-09-16T13:50:29.854 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.86199.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118199.log 2024-09-16T13:50:29.854 INFO:teuthology.orchestra.run.smithi172.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.143589.log.gz 2024-09-16T13:50:29.854 INFO:teuthology.orchestra.run.smithi172.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.73605.log.gz 2024-09-16T13:50:29.854 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126581.log 2024-09-16T13:50:29.856 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118199.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52104.log 2024-09-16T13:50:29.856 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126581.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61017.log 2024-09-16T13:50:29.856 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.52104.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.52104.log.gz 2024-09-16T13:50:29.856 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135927.log 2024-09-16T13:50:29.857 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.61017.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124560.log 2024-09-16T13:50:29.857 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135927.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50346.log 2024-09-16T13:50:29.857 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124560.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137901.log 2024-09-16T13:50:29.858 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.50346.log: 19.0% -- replaced with /var/log/ceph/cluster1-client.admin.109761.log.gz 2024-09-16T13:50:29.858 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62448.log 2024-09-16T13:50:29.859 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137901.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29015.log 2024-09-16T13:50:29.859 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.62448.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118981.log 2024-09-16T13:50:29.859 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29015.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110815.log 2024-09-16T13:50:29.860 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118981.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99395.log.gz 2024-09-16T13:50:29.860 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106718.log 2024-09-16T13:50:29.860 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.110815.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110815.log.gz 2024-09-16T13:50:29.861 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113300.log 2024-09-16T13:50:29.861 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106718.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93209.log 2024-09-16T13:50:29.861 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113300.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76884.log 2024-09-16T13:50:29.863 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93209.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135022.log 2024-09-16T13:50:29.863 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115654.log.gz 2024-09-16T13:50:29.863 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76884.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120649.log 2024-09-16T13:50:29.863 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135022.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120497.log 2024-09-16T13:50:29.865 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.120649.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30053.log 2024-09-16T13:50:29.865 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111831.log.gz 2024-09-16T13:50:29.865 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.120497.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54508.log 2024-09-16T13:50:29.865 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30053.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79875.log 2024-09-16T13:50:29.865 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54508.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88961.log 2024-09-16T13:50:29.865 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79875.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99157.log 2024-09-16T13:50:29.867 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88961.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135926.log 2024-09-16T13:50:29.867 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99157.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120743.log 2024-09-16T13:50:29.867 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.119492.log.gz 2024-09-16T13:50:29.867 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135926.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119626.log 2024-09-16T13:50:29.867 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.120743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132766.log 2024-09-16T13:50:29.867 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.119626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94312.log 2024-09-16T13:50:29.870 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132766.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47896.log 2024-09-16T13:50:29.870 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.94312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90895.log 2024-09-16T13:50:29.870 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.47896.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136010.log 2024-09-16T13:50:29.870 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125613.log 2024-09-16T13:50:29.870 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127990.log.gz 2024-09-16T13:50:29.870 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136010.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100367.log 2024-09-16T13:50:29.870 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125613.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83615.log 2024-09-16T13:50:29.873 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100367.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67787.log 2024-09-16T13:50:29.873 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.83615.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.114351.log.gz 2024-09-16T13:50:29.873 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33715.log 2024-09-16T13:50:29.873 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67787.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107529.log 2024-09-16T13:50:29.873 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32363.log 2024-09-16T13:50:29.873 INFO:teuthology.orchestra.run.smithi172.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.119834.log.gz 2024-09-16T13:50:29.873 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107529.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135719.log 2024-09-16T13:50:29.874 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.107529.log.gz 2024-09-16T13:50:29.874 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32363.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65942.log 2024-09-16T13:50:29.874 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135719.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63717.log 2024-09-16T13:50:29.875 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65942.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49142.log 2024-09-16T13:50:29.875 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100391.log 2024-09-16T13:50:29.875 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49142.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81096.log 2024-09-16T13:50:29.875 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100391.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88531.log 2024-09-16T13:50:29.875 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81096.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111344.log 2024-09-16T13:50:29.877 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88531.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77880.log 2024-09-16T13:50:29.878 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63230.log.gz 2024-09-16T13:50:29.878 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63256.log.gz 2024-09-16T13:50:29.878 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111344.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64447.log 2024-09-16T13:50:29.878 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134154.log 2024-09-16T13:50:29.878 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64447.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45909.log 2024-09-16T13:50:29.878 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134154.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81353.log 2024-09-16T13:50:29.881 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45909.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125446.log 2024-09-16T13:50:29.881 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81353.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96266.log 2024-09-16T13:50:29.881 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125446.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61582.log 2024-09-16T13:50:29.881 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96266.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28281.log 2024-09-16T13:50:29.881 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96266.log.gz 2024-09-16T13:50:29.881 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.61582.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85537.log 2024-09-16T13:50:29.881 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.28281.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28281.log.gz 2024-09-16T13:50:29.881 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79033.log 2024-09-16T13:50:29.882 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85537.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85537.log.gz 2024-09-16T13:50:29.882 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85448.log 2024-09-16T13:50:29.882 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125106.log 2024-09-16T13:50:29.884 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85448.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62421.log 2024-09-16T13:50:29.884 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125106.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95214.log 2024-09-16T13:50:29.884 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.62421.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54088.log 2024-09-16T13:50:29.884 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124056.log.gz 2024-09-16T13:50:29.884 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.95214.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53123.log 2024-09-16T13:50:29.884 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115232.log 2024-09-16T13:50:29.885 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53123.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141101.log 2024-09-16T13:50:29.885 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115232.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111154.log 2024-09-16T13:50:29.886 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103126.log 2024-09-16T13:50:29.886 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111154.log: 87.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.120424.log.gz 2024-09-16T13:50:29.886 INFO:teuthology.orchestra.run.smithi172.stderr:-5 --verbose -- /var/log/ceph/cluster1-client.mirror.121747.log 2024-09-16T13:50:29.887 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103126.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50749.log 2024-09-16T13:50:29.887 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121747.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50890.log 2024-09-16T13:50:29.887 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.50749.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138531.log 2024-09-16T13:50:29.887 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.50890.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38910.log 2024-09-16T13:50:29.888 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138531.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107549.log 2024-09-16T13:50:29.888 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.38910.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78092.log 2024-09-16T13:50:29.889 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107549.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.141940.log.gz 2024-09-16T13:50:29.890 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30562.log 2024-09-16T13:50:29.890 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78092.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108115.log 2024-09-16T13:50:29.891 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46866.log 2024-09-16T13:50:29.891 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108115.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135022.log.gz 2024-09-16T13:50:29.891 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124352.log 2024-09-16T13:50:29.891 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46866.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116623.log 2024-09-16T13:50:29.891 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124352.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133554.log 2024-09-16T13:50:29.893 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116623.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28124.log 2024-09-16T13:50:29.893 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.133554.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86287.log 2024-09-16T13:50:29.893 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133554.log.gz 2024-09-16T13:50:29.893 INFO:teuthology.orchestra.run.smithi172.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.132926.log.gz 2024-09-16T13:50:29.893 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.28124.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28124.log.gz 2024-09-16T13:50:29.893 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97202.log 2024-09-16T13:50:29.898 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.86287.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97517.log 2024-09-16T13:50:29.898 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.97202.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100032.log 2024-09-16T13:50:29.898 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.97517.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54655.log 2024-09-16T13:50:29.898 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100032.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108429.log 2024-09-16T13:50:29.898 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34556.log 2024-09-16T13:50:29.898 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.108429.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121313.log 2024-09-16T13:50:29.898 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34556.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113360.log 2024-09-16T13:50:29.898 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121313.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96719.log 2024-09-16T13:50:29.899 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113360.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104744.log.gz 2024-09-16T13:50:29.899 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60945.log 2024-09-16T13:50:29.899 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.96719.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89216.log 2024-09-16T13:50:29.899 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.60945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63846.log 2024-09-16T13:50:29.899 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89216.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60524.log 2024-09-16T13:50:29.901 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.63846.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98721.log 2024-09-16T13:50:29.901 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.60524.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116645.log 2024-09-16T13:50:29.901 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.98721.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138240.log 2024-09-16T13:50:29.901 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98721.log.gz 2024-09-16T13:50:29.901 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116645.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136455.log 2024-09-16T13:50:29.901 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138240.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109738.log 2024-09-16T13:50:29.902 INFO:teuthology.orchestra.run.smithi172.stderr: 55.8% -- replaced with /var/log/ceph/cluster2-client.mirror.27006.log.gz 2024-09-16T13:50:29.902 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136455.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104467.log.gz 2024-09-16T13:50:29.902 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89382.log 2024-09-16T13:50:29.905 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109738.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74867.log 2024-09-16T13:50:29.905 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54656.log 2024-09-16T13:50:29.905 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.74867.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109627.log 2024-09-16T13:50:29.905 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54656.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86401.log 2024-09-16T13:50:29.905 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.109627.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122541.log 2024-09-16T13:50:29.905 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.86401.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139977.log 2024-09-16T13:50:29.905 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80044.log 2024-09-16T13:50:29.905 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.139977.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44113.log 2024-09-16T13:50:29.906 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.139977.log.gz 2024-09-16T13:50:29.907 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80044.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103545.log.gz 2024-09-16T13:50:29.907 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108999.log 2024-09-16T13:50:29.907 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44113.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61790.log 2024-09-16T13:50:29.907 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.108999.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144649.log 2024-09-16T13:50:29.907 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.61790.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74573.log 2024-09-16T13:50:29.907 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.144649.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144649.log.gz 2024-09-16T13:50:29.910 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119907.log 2024-09-16T13:50:29.910 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.74573.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111790.log 2024-09-16T13:50:29.910 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.119907.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126160.log 2024-09-16T13:50:29.910 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111790.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96701.log 2024-09-16T13:50:29.910 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29225.log.gz 2024-09-16T13:50:29.910 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126160.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80086.log 2024-09-16T13:50:29.910 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.36831.log.gz 2024-09-16T13:50:29.910 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96701.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96701.log.gz 2024-09-16T13:50:29.910 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32570.log 2024-09-16T13:50:29.912 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80086.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51615.log 2024-09-16T13:50:29.912 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32570.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49817.log 2024-09-16T13:50:29.912 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.51615.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78906.log 2024-09-16T13:50:29.912 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51615.log.gz 2024-09-16T13:50:29.912 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49817.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49817.log.gz 2024-09-16T13:50:29.912 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31522.log.gz 2024-09-16T13:50:29.913 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31521.log.gz 2024-09-16T13:50:29.913 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132810.log 2024-09-16T13:50:29.914 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110908.log 2024-09-16T13:50:29.914 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.132810.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50615.log 2024-09-16T13:50:29.914 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48117.log 2024-09-16T13:50:29.916 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.50615.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141589.log 2024-09-16T13:50:29.917 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.48117.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63455.log 2024-09-16T13:50:29.917 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141589.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110125.log 2024-09-16T13:50:29.917 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39485.log.gz 2024-09-16T13:50:29.917 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.63455.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62246.log 2024-09-16T13:50:29.917 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110125.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103437.log 2024-09-16T13:50:29.917 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33715.log.gz 2024-09-16T13:50:29.918 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.62246.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66657.log 2024-09-16T13:50:29.922 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91399.log 2024-09-16T13:50:29.922 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66657.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58502.log 2024-09-16T13:50:29.922 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91399.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40323.log 2024-09-16T13:50:29.922 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58502.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136222.log 2024-09-16T13:50:29.922 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99481.log 2024-09-16T13:50:29.922 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster2-client.admin.61059.log.gz 2024-09-16T13:50:29.922 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41973.log 2024-09-16T13:50:29.922 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99481.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36288.log 2024-09-16T13:50:29.923 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41973.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101380.log 2024-09-16T13:50:29.923 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36288.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109625.log 2024-09-16T13:50:29.923 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32992.log 2024-09-16T13:50:29.923 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.109625.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.109625.log.gz 2024-09-16T13:50:29.923 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82213.log 2024-09-16T13:50:29.923 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77153.log 2024-09-16T13:50:29.923 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.82213.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.82213.log.gz 2024-09-16T13:50:29.926 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37993.log 2024-09-16T13:50:29.926 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77153.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125741.log.gz 2024-09-16T13:50:29.926 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64322.log 2024-09-16T13:50:29.926 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37993.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37993.log.gz 2024-09-16T13:50:29.926 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51730.log 2024-09-16T13:50:29.926 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.64322.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30533.log 2024-09-16T13:50:29.926 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51730.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133848.log 2024-09-16T13:50:29.926 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30533.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54721.log 2024-09-16T13:50:29.926 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133848.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125400.log 2024-09-16T13:50:29.926 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54721.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88635.log 2024-09-16T13:50:29.928 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125400.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33837.log.gz 2024-09-16T13:50:29.929 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60309.log 2024-09-16T13:50:29.929 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88635.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43693.log 2024-09-16T13:50:29.929 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.60309.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142648.log 2024-09-16T13:50:29.929 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43693.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76110.log 2024-09-16T13:50:29.929 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142648.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69730.log 2024-09-16T13:50:29.929 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76110.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143293.log 2024-09-16T13:50:29.929 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69730.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91400.log 2024-09-16T13:50:29.931 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.143293.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134082.log 2024-09-16T13:50:29.931 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91400.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92239.log 2024-09-16T13:50:29.931 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134082.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67496.log 2024-09-16T13:50:29.931 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92239.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28644.log 2024-09-16T13:50:29.931 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67496.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141790.log 2024-09-16T13:50:29.933 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27551.log 2024-09-16T13:50:29.933 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.141790.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139055.log 2024-09-16T13:50:29.933 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.27551.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115233.log 2024-09-16T13:50:29.933 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139055.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121359.log 2024-09-16T13:50:29.933 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115233.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143729.log 2024-09-16T13:50:29.934 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121359.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101965.log 2024-09-16T13:50:29.936 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.143729.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115926.log 2024-09-16T13:50:29.936 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101965.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.143729.log.gz 2024-09-16T13:50:29.936 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142639.log 2024-09-16T13:50:29.936 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115926.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143045.log 2024-09-16T13:50:29.936 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142639.log: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.143293.log.gz 2024-09-16T13:50:29.936 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109496.log 2024-09-16T13:50:29.936 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34556.log.gz 2024-09-16T13:50:29.937 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.143045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80588.log 2024-09-16T13:50:29.937 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.143045.log.gz 2024-09-16T13:50:29.938 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.109496.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133956.log 2024-09-16T13:50:29.938 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80588.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85649.log 2024-09-16T13:50:29.938 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133956.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134580.log 2024-09-16T13:50:29.938 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.85649.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66238.log 2024-09-16T13:50:29.938 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134580.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.83839.log.gz 2024-09-16T13:50:29.940 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121358.log 2024-09-16T13:50:29.940 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66238.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124984.log 2024-09-16T13:50:29.940 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43653.log.gz 2024-09-16T13:50:29.940 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.121358.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48118.log 2024-09-16T13:50:29.940 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124984.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44971.log.gz 2024-09-16T13:50:29.941 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136533.log 2024-09-16T13:50:29.942 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.48118.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37251.log 2024-09-16T13:50:29.942 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136533.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32739.log.gz 2024-09-16T13:50:29.942 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132277.log 2024-09-16T13:50:29.942 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37251.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52918.log 2024-09-16T13:50:29.943 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132277.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54801.log 2024-09-16T13:50:29.943 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52918.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97570.log 2024-09-16T13:50:29.943 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54801.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135549.log 2024-09-16T13:50:29.944 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.97570.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72563.log 2024-09-16T13:50:29.944 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135549.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45382.log 2024-09-16T13:50:29.945 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.72563.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30769.log.gz 2024-09-16T13:50:29.946 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36606.log 2024-09-16T13:50:29.946 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56609.log 2024-09-16T13:50:29.947 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36606.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109294.log 2024-09-16T13:50:29.947 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56609.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113403.log 2024-09-16T13:50:29.947 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44114.log 2024-09-16T13:50:29.948 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113403.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118937.log 2024-09-16T13:50:29.948 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44114.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85063.log 2024-09-16T13:50:29.949 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118937.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.118937.log.gz 2024-09-16T13:50:29.949 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45383.log.gz 2024-09-16T13:50:29.949 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86260.log 2024-09-16T13:50:29.951 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85063.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85063.log.gz 2024-09-16T13:50:29.951 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49812.log 2024-09-16T13:50:29.951 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.86260.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103547.log 2024-09-16T13:50:29.951 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49812.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133514.log 2024-09-16T13:50:29.951 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49812.log.gz 2024-09-16T13:50:29.952 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103547.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72543.log 2024-09-16T13:50:29.952 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41656.log.gz 2024-09-16T13:50:29.952 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58292.log 2024-09-16T13:50:29.952 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.72543.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127908.log 2024-09-16T13:50:29.953 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58292.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112395.log 2024-09-16T13:50:29.954 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90056.log 2024-09-16T13:50:29.954 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112395.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123873.log 2024-09-16T13:50:29.954 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90056.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40321.log.gz 2024-09-16T13:50:29.954 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40359.log.gz 2024-09-16T13:50:29.955 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56670.log 2024-09-16T13:50:29.955 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123873.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109770.log 2024-09-16T13:50:29.956 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56670.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79832.log 2024-09-16T13:50:29.957 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.109770.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132858.log 2024-09-16T13:50:29.957 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79832.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55142.log 2024-09-16T13:50:29.958 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132858.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.50346.log.gz 2024-09-16T13:50:29.958 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133046.log 2024-09-16T13:50:29.959 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55142.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95524.log 2024-09-16T13:50:29.959 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133046.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46601.log 2024-09-16T13:50:29.959 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.95524.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126369.log 2024-09-16T13:50:29.960 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37547.log 2024-09-16T13:50:29.960 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126369.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121680.log 2024-09-16T13:50:29.960 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54065.log.gz 2024-09-16T13:50:29.961 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37547.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118200.log 2024-09-16T13:50:29.961 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121680.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93798.log 2024-09-16T13:50:29.962 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118200.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101023.log 2024-09-16T13:50:29.962 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93798.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132258.log 2024-09-16T13:50:29.963 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101023.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.59553.log.gz 2024-09-16T13:50:29.963 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64739.log 2024-09-16T13:50:29.964 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91145.log 2024-09-16T13:50:29.964 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64848.log 2024-09-16T13:50:29.965 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91145.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88962.log 2024-09-16T13:50:29.965 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64848.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126221.log 2024-09-16T13:50:29.965 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88962.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36832.log 2024-09-16T13:50:29.965 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108858.log 2024-09-16T13:50:29.967 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.36832.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40148.log 2024-09-16T13:50:29.967 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.108858.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28550.log 2024-09-16T13:50:29.967 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.40148.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80463.log 2024-09-16T13:50:29.967 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28550.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79561.log 2024-09-16T13:50:29.968 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80463.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58335.log.gz 2024-09-16T13:50:29.968 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74108.log 2024-09-16T13:50:29.968 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79561.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37711.log 2024-09-16T13:50:29.970 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.74108.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82209.log 2024-09-16T13:50:29.970 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37711.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103857.log 2024-09-16T13:50:29.970 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54721.log.gz 2024-09-16T13:50:29.970 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.82209.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.82209.log.gz 2024-09-16T13:50:29.970 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44702.log 2024-09-16T13:50:29.974 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103857.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124479.log 2024-09-16T13:50:29.974 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105625.log 2024-09-16T13:50:29.974 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124479.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49719.log 2024-09-16T13:50:29.974 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.105625.log: gzip 0.0%-5 -- replaced with /var/log/ceph/cluster1-client.admin.105625.log.gz--verbose 2024-09-16T13:50:29.975 INFO:teuthology.orchestra.run.smithi172.stderr:-- /var/log/ceph/cluster2-client.admin.61229.log 2024-09-16T13:50:29.975 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49719.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123976.log 2024-09-16T13:50:29.975 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61229.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31100.log 2024-09-16T13:50:29.975 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123976.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85412.log 2024-09-16T13:50:29.975 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31100.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133647.log 2024-09-16T13:50:29.975 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85412.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74741.log 2024-09-16T13:50:29.977 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133647.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77153.log.gz 2024-09-16T13:50:29.977 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33369.log 2024-09-16T13:50:29.977 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.74741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62717.log 2024-09-16T13:50:29.977 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33369.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91987.log 2024-09-16T13:50:29.977 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.62717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84296.log 2024-09-16T13:50:29.977 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91987.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105163.log 2024-09-16T13:50:29.978 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.84296.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37538.log 2024-09-16T13:50:29.978 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105163.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80044.log.gz 2024-09-16T13:50:29.978 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.80086.log.gz 2024-09-16T13:50:29.978 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141251.log 2024-09-16T13:50:29.980 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37538.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80168.log.gz 2024-09-16T13:50:29.980 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80718.log 2024-09-16T13:50:29.980 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141251.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129704.log 2024-09-16T13:50:29.980 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80718.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131574.log 2024-09-16T13:50:29.982 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129704.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70329.log 2024-09-16T13:50:29.982 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.131574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126981.log 2024-09-16T13:50:29.982 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.130306.log.gz 2024-09-16T13:50:29.982 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.70329.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.70329.log.gz 2024-09-16T13:50:29.982 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115149.log 2024-09-16T13:50:29.982 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126981.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100367.log.gz 2024-09-16T13:50:29.984 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139797.log 2024-09-16T13:50:29.984 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115149.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92157.log 2024-09-16T13:50:29.984 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139797.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104909.log 2024-09-16T13:50:29.984 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92157.log: gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-09-16T13:50:29.985 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104909.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55432.log 2024-09-16T13:50:29.994 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42124.log 2024-09-16T13:50:29.994 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55432.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51927.log 2024-09-16T13:50:29.994 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42124.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46541.log 2024-09-16T13:50:29.994 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.51927.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140015.log 2024-09-16T13:50:29.994 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99887.log 2024-09-16T13:50:29.994 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.140015.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101020.log 2024-09-16T13:50:29.994 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99887.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71507.log 2024-09-16T13:50:29.994 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103547.log.gz 2024-09-16T13:50:29.994 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101020.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34634.log 2024-09-16T13:50:29.994 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71507.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110562.log 2024-09-16T13:50:29.995 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34634.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26850.log 2024-09-16T13:50:29.995 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101023.log.gz 2024-09-16T13:50:29.995 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139164.log 2024-09-16T13:50:29.995 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.26850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78612.log 2024-09-16T13:50:29.995 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139164.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40840.log 2024-09-16T13:50:29.995 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78612.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60254.log 2024-09-16T13:50:29.995 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81096.log.gz 2024-09-16T13:50:29.995 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40840.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30429.log 2024-09-16T13:50:29.995 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.60254.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119060.log 2024-09-16T13:50:29.995 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30429.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38522.log 2024-09-16T13:50:29.995 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.119060.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105120.log.gz 2024-09-16T13:50:29.995 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105163.log.gz 2024-09-16T13:50:29.996 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32359.log 2024-09-16T13:50:29.999 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.38522.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121448.log 2024-09-16T13:50:29.999 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32359.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43756.log 2024-09-16T13:50:29.999 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121448.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118634.log 2024-09-16T13:50:29.999 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58692.log 2024-09-16T13:50:29.999 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118634.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77982.log 2024-09-16T13:50:29.999 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42752.log 2024-09-16T13:50:29.999 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77982.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111154.log.gz 2024-09-16T13:50:29.999 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77460.log 2024-09-16T13:50:30.000 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42752.log: 19.0% -- replaced with /var/log/ceph/cluster2-client.admin.142551.log.gz 2024-09-16T13:50:30.000 INFO:teuthology.orchestra.run.smithi172.stderr: 19.0% -- replaced with /var/log/ceph/cluster2-client.admin.142902.log.gz 2024-09-16T13:50:30.000 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90979.log 2024-09-16T13:50:30.000 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77460.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142944.log 2024-09-16T13:50:30.000 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90979.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75498.log 2024-09-16T13:50:30.003 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142944.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32156.log 2024-09-16T13:50:30.003 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142944.log.gz 2024-09-16T13:50:30.003 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75498.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120269.log 2024-09-16T13:50:30.003 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32156.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132963.log 2024-09-16T13:50:30.003 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.120269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111156.log 2024-09-16T13:50:30.003 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.132963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88847.log 2024-09-16T13:50:30.003 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111156.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109578.log 2024-09-16T13:50:30.003 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88847.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106343.log 2024-09-16T13:50:30.004 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109578.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133356.log 2024-09-16T13:50:30.004 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106343.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137692.log 2024-09-16T13:50:30.005 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133356.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120981.log 2024-09-16T13:50:30.006 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112461.log 2024-09-16T13:50:30.006 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.120981.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130746.log 2024-09-16T13:50:30.006 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112461.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28369.log 2024-09-16T13:50:30.006 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.130746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28708.log 2024-09-16T13:50:30.006 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28369.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132611.log 2024-09-16T13:50:30.007 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28708.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117021.log 2024-09-16T13:50:30.007 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.132611.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.132611.log.gz 2024-09-16T13:50:30.007 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77671.log 2024-09-16T13:50:30.007 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117021.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.117021.log.gz 2024-09-16T13:50:30.007 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52330.log 2024-09-16T13:50:30.009 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77671.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78570.log.gz 2024-09-16T13:50:30.009 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66737.log 2024-09-16T13:50:30.011 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52330.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135128.log 2024-09-16T13:50:30.011 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66737.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134495.log 2024-09-16T13:50:30.011 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135128.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36293.log 2024-09-16T13:50:30.012 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134495.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100159.log 2024-09-16T13:50:30.012 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36293.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50138.log 2024-09-16T13:50:30.013 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100159.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73380.log 2024-09-16T13:50:30.014 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.50138.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135926.log.gz 2024-09-16T13:50:30.014 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39092.log 2024-09-16T13:50:30.014 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73380.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73380.log.gz 2024-09-16T13:50:30.014 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118087.log 2024-09-16T13:50:30.015 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.39092.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74782.log 2024-09-16T13:50:30.015 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.118087.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96350.log 2024-09-16T13:50:30.015 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133416.log.gz 2024-09-16T13:50:30.016 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.74782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113235.log 2024-09-16T13:50:30.016 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96350.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36479.log 2024-09-16T13:50:30.017 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113235.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40811.log 2024-09-16T13:50:30.018 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36479.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94460.log 2024-09-16T13:50:30.018 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40811.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62894.log 2024-09-16T13:50:30.018 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.94460.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136222.log.gz 2024-09-16T13:50:30.018 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90810.log 2024-09-16T13:50:30.022 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.62894.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111892.log 2024-09-16T13:50:30.022 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90810.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87709.log 2024-09-16T13:50:30.022 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111892.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142540.log 2024-09-16T13:50:30.023 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.87709.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87709.log.gz 2024-09-16T13:50:30.023 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88702.log 2024-09-16T13:50:30.023 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142540.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52061.log 2024-09-16T13:50:30.023 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137692.log.gz 2024-09-16T13:50:30.023 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88702.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112143.log 2024-09-16T13:50:30.023 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52061.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132819.log 2024-09-16T13:50:30.023 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64983.log 2024-09-16T13:50:30.023 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.114409.log.gz 2024-09-16T13:50:30.023 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.132819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137331.log 2024-09-16T13:50:30.023 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64983.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103986.log 2024-09-16T13:50:30.026 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89130.log 2024-09-16T13:50:30.026 INFO:teuthology.orchestra.run.smithi172.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.119907.log.gz 2024-09-16T13:50:30.026 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103986.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90012.log 2024-09-16T13:50:30.026 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89130.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104910.log 2024-09-16T13:50:30.026 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85918.log 2024-09-16T13:50:30.026 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104910.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33925.log 2024-09-16T13:50:30.026 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.85918.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129670.log 2024-09-16T13:50:30.026 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33925.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41088.log 2024-09-16T13:50:30.028 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129670.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36266.log 2024-09-16T13:50:30.028 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41088.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26928.log 2024-09-16T13:50:30.029 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36266.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79350.log 2024-09-16T13:50:30.029 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.26928.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.26928.log.gz 2024-09-16T13:50:30.029 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46119.log 2024-09-16T13:50:30.029 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79350.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142978.log 2024-09-16T13:50:30.029 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136455.log.gz 2024-09-16T13:50:30.030 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46119.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142911.log 2024-09-16T13:50:30.030 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122750.log.gz 2024-09-16T13:50:30.030 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142978.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142978.log.gz 2024-09-16T13:50:30.031 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139732.log 2024-09-16T13:50:30.031 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142911.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99654.log 2024-09-16T13:50:30.031 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139732.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38776.log 2024-09-16T13:50:30.032 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99654.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.139164.log.gz 2024-09-16T13:50:30.032 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99559.log 2024-09-16T13:50:30.032 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.38776.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45065.log 2024-09-16T13:50:30.032 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99559.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142576.log 2024-09-16T13:50:30.034 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45065.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25566.log 2024-09-16T13:50:30.034 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142576.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84887.log 2024-09-16T13:50:30.034 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113300.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.113360.log.gz 2024-09-16T13:50:30.034 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:30.034 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.25566.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25566.log.gz 2024-09-16T13:50:30.034 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114032.log 2024-09-16T13:50:30.036 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.84887.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100599.log 2024-09-16T13:50:30.036 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114032.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40387.log 2024-09-16T13:50:30.036 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100599.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61119.log 2024-09-16T13:50:30.036 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138240.log.gz 2024-09-16T13:50:30.037 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40387.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51284.log 2024-09-16T13:50:30.037 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61119.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51666.log 2024-09-16T13:50:30.038 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51284.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123931.log 2024-09-16T13:50:30.038 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51666.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126160.log.gz 2024-09-16T13:50:30.039 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.126221.log.gz 2024-09-16T13:50:30.039 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67625.log 2024-09-16T13:50:30.039 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123931.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124874.log 2024-09-16T13:50:30.039 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67625.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98927.log 2024-09-16T13:50:30.041 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124874.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31519.log 2024-09-16T13:50:30.041 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126369.log.gz 2024-09-16T13:50:30.041 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98927.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109427.log 2024-09-16T13:50:30.041 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31519.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126981.log.gz 2024-09-16T13:50:30.041 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76823.log 2024-09-16T13:50:30.041 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109427.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74781.log 2024-09-16T13:50:30.041 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76823.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85404.log 2024-09-16T13:50:30.044 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.74781.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85053.log 2024-09-16T13:50:30.044 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85404.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85404.log.gz 2024-09-16T13:50:30.044 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68890.log 2024-09-16T13:50:30.044 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85053.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88657.log 2024-09-16T13:50:30.044 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85053.log.gz 2024-09-16T13:50:30.045 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53332.log.gz 2024-09-16T13:50:30.045 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68890.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76675.log 2024-09-16T13:50:30.045 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88657.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122674.log 2024-09-16T13:50:30.045 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76675.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100118.log 2024-09-16T13:50:30.045 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122674.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96083.log 2024-09-16T13:50:30.048 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100118.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85567.log 2024-09-16T13:50:30.048 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.96083.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56822.log 2024-09-16T13:50:30.048 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85567.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86696.log 2024-09-16T13:50:30.048 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85567.log.gz 2024-09-16T13:50:30.048 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124984.log.gz 2024-09-16T13:50:30.048 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56822.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142183.log 2024-09-16T13:50:30.048 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.86696.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28174.log 2024-09-16T13:50:30.048 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142183.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125106.log.gz 2024-09-16T13:50:30.049 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126033.log 2024-09-16T13:50:30.049 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28174.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134580.log.gz 2024-09-16T13:50:30.049 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100579.log 2024-09-16T13:50:30.052 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126033.log: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.142540.log.gz 2024-09-16T13:50:30.052 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133259.log 2024-09-16T13:50:30.052 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.133356.log.gz 2024-09-16T13:50:30.052 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.133514.log.gz 2024-09-16T13:50:30.052 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100579.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56064.log 2024-09-16T13:50:30.052 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47893.log 2024-09-16T13:50:30.052 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56064.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32109.log 2024-09-16T13:50:30.052 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.47893.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133448.log 2024-09-16T13:50:30.052 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32109.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105603.log 2024-09-16T13:50:30.052 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133448.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97383.log 2024-09-16T13:50:30.053 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.105603.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105603.log.gz 2024-09-16T13:50:30.053 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62708.log 2024-09-16T13:50:30.056 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.97383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100262.log 2024-09-16T13:50:30.056 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.62708.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79142.log 2024-09-16T13:50:30.056 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100262.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93631.log 2024-09-16T13:50:30.056 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79142.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80505.log 2024-09-16T13:50:30.056 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93631.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74476.log 2024-09-16T13:50:30.056 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80505.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116139.log 2024-09-16T13:50:30.056 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.74476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110259.log 2024-09-16T13:50:30.056 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116139.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26972.log 2024-09-16T13:50:30.059 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.110259.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61904.log 2024-09-16T13:50:30.059 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.26972.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99326.log 2024-09-16T13:50:30.060 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.61904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91820.log 2024-09-16T13:50:30.060 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.99326.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143676.log 2024-09-16T13:50:30.060 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91820.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133647.log.gz 2024-09-16T13:50:30.060 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111366.log 2024-09-16T13:50:30.060 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.143676.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58693.log 2024-09-16T13:50:30.060 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.143676.log.gz 2024-09-16T13:50:30.060 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76161.log 2024-09-16T13:50:30.063 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58693.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97155.log 2024-09-16T13:50:30.063 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.76161.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49792.log 2024-09-16T13:50:30.064 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.97155.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75565.log 2024-09-16T13:50:30.064 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.129670.log.gz 2024-09-16T13:50:30.064 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49792.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49792.log.gz 2024-09-16T13:50:30.064 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127349.log 2024-09-16T13:50:30.064 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.75565.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75565.log.gz 2024-09-16T13:50:30.064 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85559.log 2024-09-16T13:50:30.064 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127349.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54235.log 2024-09-16T13:50:30.064 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85559.log: 0.0%gzip -- replaced with /var/log/ceph/cluster2-client.admin.85559.log.gz-5 2024-09-16T13:50:30.064 INFO:teuthology.orchestra.run.smithi172.stderr:--verbose -- /var/log/ceph/cluster1-client.mirror.138447.log 2024-09-16T13:50:30.064 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54235.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85065.log 2024-09-16T13:50:30.067 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138447.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125401.log 2024-09-16T13:50:30.068 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125739.log.gz 2024-09-16T13:50:30.068 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85065.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85065.log.gz 2024-09-16T13:50:30.068 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54089.log 2024-09-16T13:50:30.068 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125401.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142177.log 2024-09-16T13:50:30.068 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64503.log 2024-09-16T13:50:30.068 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142177.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82271.log 2024-09-16T13:50:30.068 INFO:teuthology.orchestra.run.smithi172.stderr: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.131574.log.gz 2024-09-16T13:50:30.068 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64503.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101755.log 2024-09-16T13:50:30.068 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.82271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100809.log 2024-09-16T13:50:30.069 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101755.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26681.log 2024-09-16T13:50:30.069 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100809.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61482.log 2024-09-16T13:50:30.074 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.26681.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133641.log 2024-09-16T13:50:30.074 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.26681.log.gz 2024-09-16T13:50:30.074 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61482.log: 0.0%gzip -- replaced with /var/log/ceph/cluster2-client.admin.61482.log.gz-5 2024-09-16T13:50:30.074 INFO:teuthology.orchestra.run.smithi172.stderr:--verbose -- /var/log/ceph/cluster1-client.mirror.109444.log 2024-09-16T13:50:30.074 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133641.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39118.log 2024-09-16T13:50:30.074 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109444.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50729.log 2024-09-16T13:50:30.075 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.39118.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144298.log 2024-09-16T13:50:30.075 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.50729.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38152.log 2024-09-16T13:50:30.075 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.63455.log.gz 2024-09-16T13:50:30.075 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.144298.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96262.log 2024-09-16T13:50:30.075 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144298.log.gz 2024-09-16T13:50:30.075 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.38152.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100912.log 2024-09-16T13:50:30.075 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96262.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120537.log 2024-09-16T13:50:30.075 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96262.log.gz 2024-09-16T13:50:30.075 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100912.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29757.log 2024-09-16T13:50:30.075 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.120537.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60765.log 2024-09-16T13:50:30.075 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29757.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137063.log 2024-09-16T13:50:30.076 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42752.log.gz 2024-09-16T13:50:30.076 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.60765.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129771.log 2024-09-16T13:50:30.082 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137063.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102595.log 2024-09-16T13:50:30.082 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.48118.log.gz 2024-09-16T13:50:30.082 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113554.log 2024-09-16T13:50:30.082 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102595.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70065.log 2024-09-16T13:50:30.082 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113554.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136643.log 2024-09-16T13:50:30.082 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.70065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64299.log 2024-09-16T13:50:30.083 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44322.log.gz 2024-09-16T13:50:30.083 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136643.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32029.log 2024-09-16T13:50:30.083 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64299.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129799.log 2024-09-16T13:50:30.083 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65106.log 2024-09-16T13:50:30.083 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27926.log 2024-09-16T13:50:30.083 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65106.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76093.log 2024-09-16T13:50:30.083 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.27926.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143843.log 2024-09-16T13:50:30.083 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54235.log.gz 2024-09-16T13:50:30.083 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76093.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121215.log 2024-09-16T13:50:30.083 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.143843.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124815.log 2024-09-16T13:50:30.083 INFO:teuthology.orchestra.run.smithi172.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.143843.log.gz 2024-09-16T13:50:30.084 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46824.log 2024-09-16T13:50:30.084 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124815.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51284.log.gz 2024-09-16T13:50:30.084 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52061.log.gz 2024-09-16T13:50:30.084 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131916.log 2024-09-16T13:50:30.089 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46824.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94284.log 2024-09-16T13:50:30.089 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.131916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37536.log 2024-09-16T13:50:30.089 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.94284.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65164.log.gz 2024-09-16T13:50:30.089 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99951.log 2024-09-16T13:50:30.089 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37536.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.144071.log 2024-09-16T13:50:30.089 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99951.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107172.log 2024-09-16T13:50:30.090 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.144071.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30679.log 2024-09-16T13:50:30.090 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.144071.log.gz 2024-09-16T13:50:30.090 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107172.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127108.log 2024-09-16T13:50:30.090 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30679.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55325.log 2024-09-16T13:50:30.090 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127108.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135087.log 2024-09-16T13:50:30.090 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92323.log 2024-09-16T13:50:30.090 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135087.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33624.log 2024-09-16T13:50:30.090 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41865.log 2024-09-16T13:50:30.090 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33624.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138069.log 2024-09-16T13:50:30.096 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41865.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140007.log 2024-09-16T13:50:30.096 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138069.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105715.log 2024-09-16T13:50:30.096 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.140007.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64022.log 2024-09-16T13:50:30.096 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105715.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60991.log 2024-09-16T13:50:30.096 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.64022.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.64022.log.gz 2024-09-16T13:50:30.096 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142525.log 2024-09-16T13:50:30.096 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.60991.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31353.log 2024-09-16T13:50:30.096 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142525.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142525.log.gz 2024-09-16T13:50:30.097 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29591.log 2024-09-16T13:50:30.097 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31353.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135803.log 2024-09-16T13:50:30.097 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29591.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33289.log 2024-09-16T13:50:30.097 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135803.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85603.log 2024-09-16T13:50:30.097 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129345.log 2024-09-16T13:50:30.097 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.85603.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105760.log 2024-09-16T13:50:30.097 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129345.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125443.log 2024-09-16T13:50:30.097 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105760.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102050.log 2024-09-16T13:50:30.101 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125443.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50206.log 2024-09-16T13:50:30.101 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102050.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92746.log 2024-09-16T13:50:30.101 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.50206.log: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.74108.log.gz 2024-09-16T13:50:30.101 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81897.log 2024-09-16T13:50:30.101 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113887.log 2024-09-16T13:50:30.101 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81897.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.86260.log.gz 2024-09-16T13:50:30.102 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90600.log 2024-09-16T13:50:30.102 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113887.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111767.log 2024-09-16T13:50:30.102 INFO:teuthology.orchestra.run.smithi172.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-client.admin.86287.log.gz 2024-09-16T13:50:30.102 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90600.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40561.log 2024-09-16T13:50:30.102 INFO:teuthology.orchestra.run.smithi172.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.75461.log.gz 2024-09-16T13:50:30.102 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111767.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46391.log 2024-09-16T13:50:30.102 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.75498.log.gz 2024-09-16T13:50:30.102 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40561.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125235.log 2024-09-16T13:50:30.106 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46391.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53460.log 2024-09-16T13:50:30.106 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125235.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117802.log 2024-09-16T13:50:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127506.log 2024-09-16T13:50:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117802.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89591.log 2024-09-16T13:50:30.107 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77460.log.gz 2024-09-16T13:50:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127506.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31191.log 2024-09-16T13:50:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89591.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57724.log 2024-09-16T13:50:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31191.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34346.log 2024-09-16T13:50:30.107 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81897.log.gz 2024-09-16T13:50:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57724.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28437.log 2024-09-16T13:50:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34346.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90056.log.gz 2024-09-16T13:50:30.107 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.144045.log 2024-09-16T13:50:30.108 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.28437.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28437.log.gz 2024-09-16T13:50:30.108 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131305.log 2024-09-16T13:50:30.112 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.144045.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130539.log 2024-09-16T13:50:30.112 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.144045.log.gz 2024-09-16T13:50:30.112 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.131305.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114180.log 2024-09-16T13:50:30.113 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130539.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114914.log 2024-09-16T13:50:30.113 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114180.log: gzip -5 --verbose -- 18.0%/var/log/ceph/cluster1-client.mirror.135654.log 2024-09-16T13:50:30.113 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.99326.log.gz 2024-09-16T13:50:30.113 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46119.log.gz 2024-09-16T13:50:30.113 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114914.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102385.log 2024-09-16T13:50:30.113 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135654.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30093.log 2024-09-16T13:50:30.113 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102385.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92282.log 2024-09-16T13:50:30.113 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30093.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54725.log 2024-09-16T13:50:30.113 INFO:teuthology.orchestra.run.smithi172.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.97155.log.gz 2024-09-16T13:50:30.113 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92282.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37999.log 2024-09-16T13:50:30.114 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54725.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88142.log 2024-09-16T13:50:30.114 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37999.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37999.log.gz 2024-09-16T13:50:30.114 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81857.log 2024-09-16T13:50:30.116 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88142.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118550.log 2024-09-16T13:50:30.116 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81857.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100913.log 2024-09-16T13:50:30.116 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.99887.log.gz 2024-09-16T13:50:30.116 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.118550.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77732.log 2024-09-16T13:50:30.116 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100913.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31399.log 2024-09-16T13:50:30.116 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77732.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61043.log 2024-09-16T13:50:30.117 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98927.log.gz 2024-09-16T13:50:30.117 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31399.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78486.log 2024-09-16T13:50:30.117 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61043.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.61043.log.gz 2024-09-16T13:50:30.117 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54302.log 2024-09-16T13:50:30.120 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78486.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110495.log 2024-09-16T13:50:30.120 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54302.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67788.log 2024-09-16T13:50:30.120 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110495.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57387.log 2024-09-16T13:50:30.120 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67788.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67494.log 2024-09-16T13:50:30.120 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57387.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83835.log 2024-09-16T13:50:30.120 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67494.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79350.log.gz 2024-09-16T13:50:30.120 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90768.log 2024-09-16T13:50:30.120 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.83835.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131963.log 2024-09-16T13:50:30.124 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90768.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139971.log 2024-09-16T13:50:30.124 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.131963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62307.log 2024-09-16T13:50:30.124 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.139971.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.139971.log.gz 2024-09-16T13:50:30.124 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114307.log 2024-09-16T13:50:30.124 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.62307.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61484.log 2024-09-16T13:50:30.124 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114307.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108455.log 2024-09-16T13:50:30.124 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61484.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.61484.log.gz 2024-09-16T13:50:30.124 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107831.log 2024-09-16T13:50:30.124 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.108455.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134962.log 2024-09-16T13:50:30.124 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55982.log 2024-09-16T13:50:30.128 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134962.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52964.log 2024-09-16T13:50:30.129 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55982.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86126.log 2024-09-16T13:50:30.129 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52964.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50916.log 2024-09-16T13:50:30.129 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68890.log.gz 2024-09-16T13:50:30.129 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.86126.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134430.log 2024-09-16T13:50:30.129 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.50916.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116348.log 2024-09-16T13:50:30.129 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134430.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29464.log 2024-09-16T13:50:30.129 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116348.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108315.log 2024-09-16T13:50:30.129 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102492.log 2024-09-16T13:50:30.129 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.108315.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109113.log 2024-09-16T13:50:30.129 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102492.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114033.log 2024-09-16T13:50:30.133 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.109113.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71736.log 2024-09-16T13:50:30.133 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85717.log 2024-09-16T13:50:30.133 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71736.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111665.log 2024-09-16T13:50:30.133 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.85717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91146.log 2024-09-16T13:50:30.133 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111665.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48238.log 2024-09-16T13:50:30.134 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91146.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56481.log 2024-09-16T13:50:30.134 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.48238.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37679.log 2024-09-16T13:50:30.134 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56481.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78720.log 2024-09-16T13:50:30.134 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37679.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112252.log 2024-09-16T13:50:30.134 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78720.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108241.log 2024-09-16T13:50:30.138 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112252.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38340.log 2024-09-16T13:50:30.138 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108241.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.108241.log.gz 2024-09-16T13:50:30.138 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63783.log 2024-09-16T13:50:30.138 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.38340.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132390.log 2024-09-16T13:50:30.138 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63783.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104700.log 2024-09-16T13:50:30.138 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104618.log 2024-09-16T13:50:30.139 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104700.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58020.log 2024-09-16T13:50:30.139 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104618.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41763.log 2024-09-16T13:50:30.139 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58020.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85629.log 2024-09-16T13:50:30.139 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41763.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102707.log 2024-09-16T13:50:30.139 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.85629.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125716.log 2024-09-16T13:50:30.143 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102707.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65056.log 2024-09-16T13:50:30.143 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36073.log 2024-09-16T13:50:30.143 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65056.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95213.log 2024-09-16T13:50:30.143 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36073.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49209.log 2024-09-16T13:50:30.143 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.95213.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76073.log 2024-09-16T13:50:30.143 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49209.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44808.log 2024-09-16T13:50:30.143 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68106.log 2024-09-16T13:50:30.143 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44808.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113299.log 2024-09-16T13:50:30.144 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68106.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85079.log 2024-09-16T13:50:30.144 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113299.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53029.log 2024-09-16T13:50:30.144 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85079.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71435.log 2024-09-16T13:50:30.147 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68257.log 2024-09-16T13:50:30.147 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71435.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126288.log 2024-09-16T13:50:30.147 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68257.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139332.log 2024-09-16T13:50:30.147 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126288.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137859.log 2024-09-16T13:50:30.147 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88326.log 2024-09-16T13:50:30.147 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137859.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130353.log 2024-09-16T13:50:30.147 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88326.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138239.log 2024-09-16T13:50:30.147 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130353.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130353.log.gz 2024-09-16T13:50:30.148 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36222.log 2024-09-16T13:50:30.150 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138239.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48965.log 2024-09-16T13:50:30.151 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116414.log 2024-09-16T13:50:30.151 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.48965.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138868.log 2024-09-16T13:50:30.151 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116414.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134751.log 2024-09-16T13:50:30.151 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138868.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56903.log 2024-09-16T13:50:30.151 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139818.log 2024-09-16T13:50:30.151 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56903.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29425.log 2024-09-16T13:50:30.151 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139818.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51819.log 2024-09-16T13:50:30.154 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29425.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40658.log 2024-09-16T13:50:30.154 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66634.log 2024-09-16T13:50:30.154 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40658.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128012.log 2024-09-16T13:50:30.154 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66634.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25773.log 2024-09-16T13:50:30.154 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.128012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76355.log 2024-09-16T13:50:30.154 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.25773.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25773.log.gz 2024-09-16T13:50:30.154 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80087.log 2024-09-16T13:50:30.154 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76355.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105229.log 2024-09-16T13:50:30.156 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80087.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75632.log 2024-09-16T13:50:30.156 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105229.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97456.log 2024-09-16T13:50:30.156 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75632.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31480.log 2024-09-16T13:50:30.156 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.97456.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50635.log 2024-09-16T13:50:30.157 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31480.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78401.log 2024-09-16T13:50:30.158 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.50635.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66214.log 2024-09-16T13:50:30.159 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78401.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33415.log 2024-09-16T13:50:30.159 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66214.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120955.log 2024-09-16T13:50:30.159 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33415.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32870.log 2024-09-16T13:50:30.159 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.120955.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116727.log 2024-09-16T13:50:30.160 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32870.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63630.log 2024-09-16T13:50:30.160 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116727.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94437.log 2024-09-16T13:50:30.161 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.63630.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66107.log 2024-09-16T13:50:30.161 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.94437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62193.log 2024-09-16T13:50:30.162 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66107.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98483.log 2024-09-16T13:50:30.163 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.62193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130427.log 2024-09-16T13:50:30.163 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98483.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106968.log 2024-09-16T13:50:30.163 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130427.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47226.log 2024-09-16T13:50:30.164 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106968.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74134.log 2024-09-16T13:50:30.165 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.47226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89341.log 2024-09-16T13:50:30.165 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.74134.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66656.log 2024-09-16T13:50:30.165 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80088.log 2024-09-16T13:50:30.167 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66656.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110856.log 2024-09-16T13:50:30.167 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101655.log 2024-09-16T13:50:30.167 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110856.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75155.log 2024-09-16T13:50:30.167 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101655.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70309.log 2024-09-16T13:50:30.167 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75155.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38817.log 2024-09-16T13:50:30.169 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.70309.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.70309.log.gz 2024-09-16T13:50:30.169 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39374.log 2024-09-16T13:50:30.169 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.38817.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119813.log 2024-09-16T13:50:30.169 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.39374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80886.log 2024-09-16T13:50:30.170 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.119813.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107538.log 2024-09-16T13:50:30.170 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80886.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106571.log 2024-09-16T13:50:30.170 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107538.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110497.log 2024-09-16T13:50:30.172 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106571.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40139.log 2024-09-16T13:50:30.172 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110497.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58792.log 2024-09-16T13:50:30.172 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.40139.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138869.log 2024-09-16T13:50:30.174 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121706.log 2024-09-16T13:50:30.174 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138869.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108683.log 2024-09-16T13:50:30.174 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121706.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76031.log 2024-09-16T13:50:30.174 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.108683.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107540.log 2024-09-16T13:50:30.176 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76031.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33082.log 2024-09-16T13:50:30.176 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107540.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113761.log 2024-09-16T13:50:30.176 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33082.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102219.log 2024-09-16T13:50:30.176 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113761.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28082.log 2024-09-16T13:50:30.176 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102219.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94938.log 2024-09-16T13:50:30.179 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28082.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131097.log 2024-09-16T13:50:30.179 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.94938.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64652.log 2024-09-16T13:50:30.179 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.131097.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131097.log.gz 2024-09-16T13:50:30.180 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67204.log 2024-09-16T13:50:30.180 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57876.log 2024-09-16T13:50:30.180 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67204.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61074.log 2024-09-16T13:50:30.180 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57876.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100493.log 2024-09-16T13:50:30.180 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61074.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95519.log 2024-09-16T13:50:30.183 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36480.log 2024-09-16T13:50:30.183 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.95519.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75834.log 2024-09-16T13:50:30.183 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36480.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39287.log 2024-09-16T13:50:30.183 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75834.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59552.log 2024-09-16T13:50:30.183 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39287.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44175.log 2024-09-16T13:50:30.183 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59552.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30893.log 2024-09-16T13:50:30.183 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44175.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73074.log 2024-09-16T13:50:30.186 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30893.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120176.log 2024-09-16T13:50:30.186 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73074.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64024.log 2024-09-16T13:50:30.186 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.120176.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129536.log 2024-09-16T13:50:30.186 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.64024.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32319.log 2024-09-16T13:50:30.186 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46971.log 2024-09-16T13:50:30.186 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32319.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123428.log 2024-09-16T13:50:30.189 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46971.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118312.log 2024-09-16T13:50:30.189 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123428.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32740.log 2024-09-16T13:50:30.189 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70315.log 2024-09-16T13:50:30.189 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32740.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57325.log 2024-09-16T13:50:30.189 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.70315.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.70315.log.gz 2024-09-16T13:50:30.190 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94075.log 2024-09-16T13:50:30.190 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115359.log 2024-09-16T13:50:30.190 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.94075.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118333.log 2024-09-16T13:50:30.192 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115359.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135760.log 2024-09-16T13:50:30.192 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62079.log 2024-09-16T13:50:30.192 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135760.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59547.log 2024-09-16T13:50:30.192 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.62079.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104323.log 2024-09-16T13:50:30.192 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59547.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46847.log 2024-09-16T13:50:30.192 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69312.log 2024-09-16T13:50:30.194 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.46847.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46847.log.gz 2024-09-16T13:50:30.194 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142191.log 2024-09-16T13:50:30.194 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69312.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139976.log 2024-09-16T13:50:30.194 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142191.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43694.log 2024-09-16T13:50:30.195 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.139976.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.139976.log.gz 2024-09-16T13:50:30.195 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102010.log 2024-09-16T13:50:30.195 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43694.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95248.log 2024-09-16T13:50:30.196 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102010.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118544.log 2024-09-16T13:50:30.197 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.95248.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125444.log 2024-09-16T13:50:30.198 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.118544.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135298.log 2024-09-16T13:50:30.198 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.133259.log.gz 2024-09-16T13:50:30.198 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125444.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128033.log 2024-09-16T13:50:30.199 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135298.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53855.log 2024-09-16T13:50:30.199 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.128033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93421.log 2024-09-16T13:50:30.200 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53855.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86675.log 2024-09-16T13:50:30.200 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93421.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112839.log 2024-09-16T13:50:30.202 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.86675.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120783.log 2024-09-16T13:50:30.202 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112839.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133218.log 2024-09-16T13:50:30.202 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.120783.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124815.log.gz 2024-09-16T13:50:30.202 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115193.log 2024-09-16T13:50:30.203 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133218.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43063.log 2024-09-16T13:50:30.203 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80294.log 2024-09-16T13:50:30.203 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43063.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141096.log 2024-09-16T13:50:30.204 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46824.log.gz 2024-09-16T13:50:30.204 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80294.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60308.log 2024-09-16T13:50:30.204 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141096.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111767.log.gz 2024-09-16T13:50:30.206 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70297.log 2024-09-16T13:50:30.206 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.60308.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30349.log 2024-09-16T13:50:30.206 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.70297.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.70297.log.gz 2024-09-16T13:50:30.206 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120537.log.gz 2024-09-16T13:50:30.206 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144206.log 2024-09-16T13:50:30.207 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30349.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54445.log 2024-09-16T13:50:30.207 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.144206.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144206.log.gz 2024-09-16T13:50:30.207 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77352.log 2024-09-16T13:50:30.209 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54445.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61509.log 2024-09-16T13:50:30.209 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77352.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114032.log.gz 2024-09-16T13:50:30.209 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114033.log.gz 2024-09-16T13:50:30.209 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58442.log 2024-09-16T13:50:30.210 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.61509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101545.log 2024-09-16T13:50:30.211 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67498.log 2024-09-16T13:50:30.212 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.114180.log.gz 2024-09-16T13:50:30.212 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27765.log 2024-09-16T13:50:30.212 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67498.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114200.log 2024-09-16T13:50:30.212 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.27765.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68680.log 2024-09-16T13:50:30.214 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114200.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65584.log 2024-09-16T13:50:30.214 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68680.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52527.log 2024-09-16T13:50:30.214 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65584.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34002.log 2024-09-16T13:50:30.214 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.52527.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.52527.log.gz 2024-09-16T13:50:30.214 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139732.log.gz 2024-09-16T13:50:30.214 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58526.log 2024-09-16T13:50:30.216 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46858.log 2024-09-16T13:50:30.216 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58526.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93336.log 2024-09-16T13:50:30.217 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137331.log.gz 2024-09-16T13:50:30.217 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.46858.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46858.log.gz 2024-09-16T13:50:30.217 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57808.log 2024-09-16T13:50:30.217 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93336.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104533.log 2024-09-16T13:50:30.219 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57808.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96686.log 2024-09-16T13:50:30.219 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104533.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130928.log 2024-09-16T13:50:30.219 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120685.log 2024-09-16T13:50:30.220 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96686.log.gz 2024-09-16T13:50:30.220 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130928.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64892.log 2024-09-16T13:50:30.220 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138868.log.gz 2024-09-16T13:50:30.220 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.120685.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62535.log 2024-09-16T13:50:30.220 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64892.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71819.log 2024-09-16T13:50:30.222 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.62535.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95759.log 2024-09-16T13:50:30.222 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.71819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135861.log 2024-09-16T13:50:30.222 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.95759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53879.log 2024-09-16T13:50:30.222 INFO:teuthology.orchestra.run.smithi172.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52330.log.gz 2024-09-16T13:50:30.222 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135861.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123641.log 2024-09-16T13:50:30.222 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53879.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119368.log 2024-09-16T13:50:30.226 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123641.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138743.log 2024-09-16T13:50:30.226 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50729.log.gz 2024-09-16T13:50:30.226 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.119368.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140006.log 2024-09-16T13:50:30.226 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119368.log.gz 2024-09-16T13:50:30.226 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138743.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96706.log 2024-09-16T13:50:30.226 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.140006.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48717.log 2024-09-16T13:50:30.226 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96706.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96706.log.gz 2024-09-16T13:50:30.226 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51976.log 2024-09-16T13:50:30.226 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67625.log.gz 2024-09-16T13:50:30.227 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.48717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125528.log 2024-09-16T13:50:30.228 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51976.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65626.log 2024-09-16T13:50:30.228 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125528.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53460.log.gz 2024-09-16T13:50:30.228 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58419.log 2024-09-16T13:50:30.229 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65626.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130411.log 2024-09-16T13:50:30.229 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58419.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94095.log 2024-09-16T13:50:30.230 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130411.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57747.log 2024-09-16T13:50:30.230 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.94095.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57325.log.gz 2024-09-16T13:50:30.230 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132808.log 2024-09-16T13:50:30.230 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57387.log.gz 2024-09-16T13:50:30.230 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57747.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124604.log 2024-09-16T13:50:30.231 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.132808.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.132808.log.gz 2024-09-16T13:50:30.232 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106684.log 2024-09-16T13:50:30.232 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33512.log 2024-09-16T13:50:30.232 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106684.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49616.log 2024-09-16T13:50:30.232 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33512.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54445.log.gz 2024-09-16T13:50:30.233 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123721.log 2024-09-16T13:50:30.234 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49616.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98265.log 2024-09-16T13:50:30.234 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123721.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135506.log 2024-09-16T13:50:30.234 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65106.log.gz 2024-09-16T13:50:30.234 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98265.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97497.log 2024-09-16T13:50:30.235 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135506.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79984.log 2024-09-16T13:50:30.236 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.97497.log: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.51927.log.gz 2024-09-16T13:50:30.236 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143904.log 2024-09-16T13:50:30.237 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79984.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98889.log 2024-09-16T13:50:30.237 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.143904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35035.log 2024-09-16T13:50:30.237 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.143904.log.gz 2024-09-16T13:50:30.237 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.98889.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101446.log 2024-09-16T13:50:30.239 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.35035.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131105.log 2024-09-16T13:50:30.239 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101446.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101335.log 2024-09-16T13:50:30.239 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.131105.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131105.log.gz 2024-09-16T13:50:30.239 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57324.log 2024-09-16T13:50:30.239 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98421.log 2024-09-16T13:50:30.240 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57324.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96283.log 2024-09-16T13:50:30.241 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65584.log.gz 2024-09-16T13:50:30.241 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98421.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34942.log 2024-09-16T13:50:30.241 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75634.log 2024-09-16T13:50:30.242 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.34942.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34942.log.gz 2024-09-16T13:50:30.242 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137541.log 2024-09-16T13:50:30.242 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44175.log.gz 2024-09-16T13:50:30.242 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75634.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105611.log 2024-09-16T13:50:30.243 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78464.log 2024-09-16T13:50:30.244 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.105611.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105611.log.gz 2024-09-16T13:50:30.244 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58442.log.gz 2024-09-16T13:50:30.244 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53135.log 2024-09-16T13:50:30.245 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78464.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58526.log.gz 2024-09-16T13:50:30.245 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141592.log 2024-09-16T13:50:30.245 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53135.log: gzip 90.0%-5 --verbose -- replaced with /var/log/ceph/cluster1-client.mirror.79984.log.gz-- 2024-09-16T13:50:30.245 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52392.log 2024-09-16T13:50:30.246 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141592.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122669.log 2024-09-16T13:50:30.246 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52392.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110533.log 2024-09-16T13:50:30.247 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122669.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81857.log.gz 2024-09-16T13:50:30.247 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111223.log 2024-09-16T13:50:30.248 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110533.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105289.log 2024-09-16T13:50:30.248 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111223.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94284.log.gz 2024-09-16T13:50:30.249 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69188.log 2024-09-16T13:50:30.249 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106758.log 2024-09-16T13:50:30.249 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69188.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53543.log 2024-09-16T13:50:30.250 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106758.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101757.log 2024-09-16T13:50:30.251 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53543.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37664.log 2024-09-16T13:50:30.251 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101757.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104257.log 2024-09-16T13:50:30.251 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37664.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137902.log 2024-09-16T13:50:30.252 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104257.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100159.log.gz 2024-09-16T13:50:30.253 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.100118.log.gz 2024-09-16T13:50:30.253 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107717.log 2024-09-16T13:50:30.254 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137902.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135442.log 2024-09-16T13:50:30.254 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107717.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.107717.log.gz 2024-09-16T13:50:30.254 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.107538.log.gz 2024-09-16T13:50:30.254 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98965.log 2024-09-16T13:50:30.255 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104069.log.gz 2024-09-16T13:50:30.257 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79982.log 2024-09-16T13:50:30.257 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98965.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60530.log 2024-09-16T13:50:30.257 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79982.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76519.log 2024-09-16T13:50:30.257 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.60530.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113760.log 2024-09-16T13:50:30.257 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76519.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80294.log.gz 2024-09-16T13:50:30.258 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28019.log 2024-09-16T13:50:30.258 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113760.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34259.log 2024-09-16T13:50:30.259 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123765.log 2024-09-16T13:50:30.259 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34259.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48288.log 2024-09-16T13:50:30.260 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123765.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106545.log 2024-09-16T13:50:30.260 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.48288.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62173.log 2024-09-16T13:50:30.262 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131138.log 2024-09-16T13:50:30.262 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.62173.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62287.log 2024-09-16T13:50:30.262 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.131138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131708.log 2024-09-16T13:50:30.262 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.62287.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126663.log 2024-09-16T13:50:30.262 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91146.log.gz 2024-09-16T13:50:30.262 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.131708.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37529.log 2024-09-16T13:50:30.264 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126663.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119052.log 2024-09-16T13:50:30.264 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37529.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37529.log.gz 2024-09-16T13:50:30.264 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86541.log 2024-09-16T13:50:30.264 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.119052.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43062.log 2024-09-16T13:50:30.265 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.86541.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72092.log 2024-09-16T13:50:30.265 INFO:teuthology.orchestra.run.smithi172.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.86696.log.gz 2024-09-16T13:50:30.268 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81643.log 2024-09-16T13:50:30.268 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.72092.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36069.log 2024-09-16T13:50:30.268 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81643.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98658.log 2024-09-16T13:50:30.268 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36069.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131113.log 2024-09-16T13:50:30.268 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98658.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108113.log 2024-09-16T13:50:30.268 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.131113.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131113.log.gz 2024-09-16T13:50:30.268 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135801.log 2024-09-16T13:50:30.269 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108113.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.108113.log.gz 2024-09-16T13:50:30.269 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57115.log.gz 2024-09-16T13:50:30.269 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69476.log 2024-09-16T13:50:30.271 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135801.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60776.log 2024-09-16T13:50:30.271 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75353.log 2024-09-16T13:50:30.272 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.60776.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55982.log.gz 2024-09-16T13:50:30.272 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112040.log 2024-09-16T13:50:30.272 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.75353.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79664.log 2024-09-16T13:50:30.272 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112040.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26226.log 2024-09-16T13:50:30.272 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116727.log.gz 2024-09-16T13:50:30.272 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79664.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.17522.log 2024-09-16T13:50:30.272 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.26226.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.26226.log.gz 2024-09-16T13:50:30.274 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38320.log 2024-09-16T13:50:30.274 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/ceph.tmp-client.admin.17522.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.17522.log.gz 2024-09-16T13:50:30.274 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106576.log 2024-09-16T13:50:30.274 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.38320.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111223.log.gz 2024-09-16T13:50:30.275 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137483.log 2024-09-16T13:50:30.275 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106576.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114770.log 2024-09-16T13:50:30.275 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137483.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37519.log 2024-09-16T13:50:30.277 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114770.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32320.log 2024-09-16T13:50:30.277 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37519.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37519.log.gz 2024-09-16T13:50:30.277 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107530.log 2024-09-16T13:50:30.277 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91607.log 2024-09-16T13:50:30.277 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107530.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112839.log.gz 2024-09-16T13:50:30.277 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.107530.log.gz 2024-09-16T13:50:30.277 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70006.log 2024-09-16T13:50:30.279 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91607.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129856.log 2024-09-16T13:50:30.280 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.70006.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139501.log 2024-09-16T13:50:30.280 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129856.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58663.log 2024-09-16T13:50:30.280 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139501.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58375.log 2024-09-16T13:50:30.280 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.58663.log: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.118087.log.gz 2024-09-16T13:50:30.280 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.58663.log.gz 2024-09-16T13:50:30.280 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103524.log 2024-09-16T13:50:30.283 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103568.log 2024-09-16T13:50:30.283 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103524.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128097.log 2024-09-16T13:50:30.283 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118544.log.gz 2024-09-16T13:50:30.283 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103568.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103226.log 2024-09-16T13:50:30.283 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.128097.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76378.log 2024-09-16T13:50:30.283 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67285.log 2024-09-16T13:50:30.283 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76378.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107949.log 2024-09-16T13:50:30.284 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67285.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57724.log.gz 2024-09-16T13:50:30.284 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41510.log 2024-09-16T13:50:30.285 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107949.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-09-16T13:50:30.286 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95116.log 2024-09-16T13:50:30.286 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138383.log 2024-09-16T13:50:30.288 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.95116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142544.log 2024-09-16T13:50:30.288 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44742.log 2024-09-16T13:50:30.288 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142544.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125863.log 2024-09-16T13:50:30.288 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44742.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27074.log 2024-09-16T13:50:30.288 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125863.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29421.log 2024-09-16T13:50:30.289 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.27074.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89217.log 2024-09-16T13:50:30.291 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29421.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125864.log 2024-09-16T13:50:30.291 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73070.log 2024-09-16T13:50:30.291 INFO:teuthology.orchestra.run.smithi172.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108455.log.gz 2024-09-16T13:50:30.291 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125864.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30772.log 2024-09-16T13:50:30.291 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73070.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107868.log 2024-09-16T13:50:30.291 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30772.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86582.log 2024-09-16T13:50:30.293 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107868.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130899.log 2024-09-16T13:50:30.293 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.86582.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33623.log 2024-09-16T13:50:30.293 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130899.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139710.log 2024-09-16T13:50:30.293 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33623.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41128.log 2024-09-16T13:50:30.294 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139710.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127717.log 2024-09-16T13:50:30.295 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139501.log.gz 2024-09-16T13:50:30.295 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41128.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111054.log 2024-09-16T13:50:30.295 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31143.log 2024-09-16T13:50:30.296 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111054.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73652.log 2024-09-16T13:50:30.296 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123721.log.gz 2024-09-16T13:50:30.296 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51442.log 2024-09-16T13:50:30.296 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.73652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141412.log 2024-09-16T13:50:30.298 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.51442.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96792.log 2024-09-16T13:50:30.299 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141412.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129289.log 2024-09-16T13:50:30.299 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.96792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.96813.log 2024-09-16T13:50:30.299 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126242.log 2024-09-16T13:50:30.299 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.96813.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73698.log 2024-09-16T13:50:30.301 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126242.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71504.log 2024-09-16T13:50:30.301 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.73698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59774.log 2024-09-16T13:50:30.301 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71504.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33922.log 2024-09-16T13:50:30.301 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59774.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76029.log 2024-09-16T13:50:30.301 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33922.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34463.log 2024-09-16T13:50:30.304 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39288.log 2024-09-16T13:50:30.304 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34463.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78990.log 2024-09-16T13:50:30.304 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39288.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104175.log 2024-09-16T13:50:30.304 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78990.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48841.log 2024-09-16T13:50:30.304 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104175.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98525.log 2024-09-16T13:50:30.304 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.48841.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110076.log 2024-09-16T13:50:30.305 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.98525.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98525.log.gz 2024-09-16T13:50:30.305 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28781.log 2024-09-16T13:50:30.307 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.110076.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50993.log 2024-09-16T13:50:30.307 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.28781.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127778.log 2024-09-16T13:50:30.308 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.50993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79411.log 2024-09-16T13:50:30.308 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127778.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116073.log 2024-09-16T13:50:30.308 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79411.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115823.log 2024-09-16T13:50:30.308 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59724.log 2024-09-16T13:50:30.308 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115823.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131667.log 2024-09-16T13:50:30.309 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59724.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64953.log 2024-09-16T13:50:30.309 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.131667.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101966.log 2024-09-16T13:50:30.309 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64953.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130223.log 2024-09-16T13:50:30.310 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101966.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55410.log 2024-09-16T13:50:30.311 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.130223.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27654.log 2024-09-16T13:50:30.311 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55410.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141415.log 2024-09-16T13:50:30.313 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.27654.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105459.log 2024-09-16T13:50:30.313 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141415.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135398.log 2024-09-16T13:50:30.313 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105459.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88491.log 2024-09-16T13:50:30.313 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87916.log 2024-09-16T13:50:30.315 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88491.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65615.log 2024-09-16T13:50:30.315 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.130411.log.gz 2024-09-16T13:50:30.315 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.87916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71899.log 2024-09-16T13:50:30.315 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65615.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53310.log 2024-09-16T13:50:30.316 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.71899.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121905.log 2024-09-16T13:50:30.316 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53310.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28553.log 2024-09-16T13:50:30.318 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121905.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25932.log 2024-09-16T13:50:30.318 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28553.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115882.log 2024-09-16T13:50:30.318 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.25932.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25932.log.gz 2024-09-16T13:50:30.318 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34464.log 2024-09-16T13:50:30.318 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122361.log 2024-09-16T13:50:30.318 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34464.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37522.log 2024-09-16T13:50:30.319 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122361.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137650.log 2024-09-16T13:50:30.319 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37522.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37522.log.gz 2024-09-16T13:50:30.320 INFO:teuthology.orchestra.run.smithi172.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.96083.log.gz 2024-09-16T13:50:30.320 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90685.log 2024-09-16T13:50:30.321 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137650.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63300.log 2024-09-16T13:50:30.321 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90685.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93546.log 2024-09-16T13:50:30.321 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.63300.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89341.log.gz 2024-09-16T13:50:30.321 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73033.log 2024-09-16T13:50:30.322 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93546.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120554.log 2024-09-16T13:50:30.323 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.73033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92788.log 2024-09-16T13:50:30.323 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.120554.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50252.log 2024-09-16T13:50:30.324 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92788.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52816.log 2024-09-16T13:50:30.324 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.50252.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.77732.log.gz 2024-09-16T13:50:30.324 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32152.log 2024-09-16T13:50:30.325 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52816.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143817.log 2024-09-16T13:50:30.326 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32152.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63309.log 2024-09-16T13:50:30.326 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.143817.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73538.log 2024-09-16T13:50:30.326 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.63309.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29970.log 2024-09-16T13:50:30.326 INFO:teuthology.orchestra.run.smithi172.stderr: 90.8% -- replaced with /var/log/ceph/cluster1-client.admin.143817.log.gz 2024-09-16T13:50:30.327 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.73538.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26021.log 2024-09-16T13:50:30.328 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29970.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114600.log 2024-09-16T13:50:30.328 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.26021.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.26021.log.gz 2024-09-16T13:50:30.328 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130303.log 2024-09-16T13:50:30.329 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114600.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49119.log 2024-09-16T13:50:30.329 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81351.log 2024-09-16T13:50:30.329 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49119.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49119.log.gz 2024-09-16T13:50:30.329 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.76073.log.gz 2024-09-16T13:50:30.331 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34684.log 2024-09-16T13:50:30.331 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29884.log 2024-09-16T13:50:30.331 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34684.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49500.log 2024-09-16T13:50:30.332 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29884.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94054.log 2024-09-16T13:50:30.332 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.49500.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89591.log.gz 2024-09-16T13:50:30.332 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142552.log 2024-09-16T13:50:30.333 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.94054.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76945.log 2024-09-16T13:50:30.333 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142552.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65898.log 2024-09-16T13:50:30.335 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90391.log 2024-09-16T13:50:30.335 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65898.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57327.log 2024-09-16T13:50:30.335 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90391.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53334.log 2024-09-16T13:50:30.336 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57327.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49115.log 2024-09-16T13:50:30.336 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53334.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94075.log.gz 2024-09-16T13:50:30.337 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.94095.log.gz 2024-09-16T13:50:30.337 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52041.log 2024-09-16T13:50:30.337 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49115.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49115.log.gz 2024-09-16T13:50:30.337 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.37538.log.gz 2024-09-16T13:50:30.338 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131895.log 2024-09-16T13:50:30.338 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.37536.log.gz 2024-09-16T13:50:30.339 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52041.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87094.log 2024-09-16T13:50:30.339 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.131895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72338.log 2024-09-16T13:50:30.341 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87094.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43803.log 2024-09-16T13:50:30.341 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.72338.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78826.log 2024-09-16T13:50:30.341 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43803.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57031.log 2024-09-16T13:50:30.341 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78826.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32871.log 2024-09-16T13:50:30.342 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57031.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46974.log 2024-09-16T13:50:30.342 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32871.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123824.log 2024-09-16T13:50:30.343 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46974.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138743.log.gz 2024-09-16T13:50:30.343 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130488.log 2024-09-16T13:50:30.343 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123824.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128982.log 2024-09-16T13:50:30.346 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130488.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142533.log 2024-09-16T13:50:30.346 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.128982.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43862.log 2024-09-16T13:50:30.346 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142533.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142533.log.gz 2024-09-16T13:50:30.346 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27514.log 2024-09-16T13:50:30.347 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135801.log.gz 2024-09-16T13:50:30.347 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43862.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111086.log 2024-09-16T13:50:30.347 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.27514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82394.log 2024-09-16T13:50:30.347 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111086.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69265.log 2024-09-16T13:50:30.349 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.82394.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40498.log 2024-09-16T13:50:30.349 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69265.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113931.log 2024-09-16T13:50:30.349 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40498.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92831.log 2024-09-16T13:50:30.349 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135442.log.gz 2024-09-16T13:50:30.349 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38320.log.gz 2024-09-16T13:50:30.349 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113931.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101210.log 2024-09-16T13:50:30.351 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68551.log 2024-09-16T13:50:30.351 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101210.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44854.log 2024-09-16T13:50:30.351 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68551.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72167.log 2024-09-16T13:50:30.352 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44854.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78317.log 2024-09-16T13:50:30.352 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.72167.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49910.log 2024-09-16T13:50:30.353 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78317.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31942.log 2024-09-16T13:50:30.354 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.49910.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79326.log 2024-09-16T13:50:30.354 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.128033.log.gz 2024-09-16T13:50:30.354 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31942.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45652.log 2024-09-16T13:50:30.356 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79326.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125821.log 2024-09-16T13:50:30.356 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98106.log 2024-09-16T13:50:30.356 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67684.log 2024-09-16T13:50:30.356 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98106.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31734.log 2024-09-16T13:50:30.356 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67684.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128915.log 2024-09-16T13:50:30.359 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31734.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45911.log 2024-09-16T13:50:30.359 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.128915.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133084.log 2024-09-16T13:50:30.359 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45911.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132335.log 2024-09-16T13:50:30.359 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113301.log 2024-09-16T13:50:30.359 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.132335.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.132335.log.gz 2024-09-16T13:50:30.360 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33832.log 2024-09-16T13:50:30.360 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113301.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56691.log 2024-09-16T13:50:30.362 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33832.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101799.log 2024-09-16T13:50:30.362 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56691.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35099.log 2024-09-16T13:50:30.362 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.28550.log.gz 2024-09-16T13:50:30.362 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110455.log 2024-09-16T13:50:30.362 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.35099.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37939.log 2024-09-16T13:50:30.363 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.110455.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71502.log 2024-09-16T13:50:30.363 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37939.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37939.log.gz 2024-09-16T13:50:30.363 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32320.log.gz 2024-09-16T13:50:30.364 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26955.log 2024-09-16T13:50:30.364 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71502.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48095.log 2024-09-16T13:50:30.364 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.26955.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26148.log 2024-09-16T13:50:30.364 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.48095.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59771.log 2024-09-16T13:50:30.366 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.26148.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.26148.log.gz 2024-09-16T13:50:30.366 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108112.log 2024-09-16T13:50:30.366 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46494.log 2024-09-16T13:50:30.366 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108112.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.108112.log.gz 2024-09-16T13:50:30.367 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100579.log.gz 2024-09-16T13:50:30.367 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55221.log 2024-09-16T13:50:30.368 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46494.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40458.log 2024-09-16T13:50:30.369 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55221.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36829.log 2024-09-16T13:50:30.369 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.40458.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53604.log 2024-09-16T13:50:30.369 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50957.log.gz 2024-09-16T13:50:30.369 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.36829.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40979.log 2024-09-16T13:50:30.369 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34985.log 2024-09-16T13:50:30.372 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40979.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42916.log 2024-09-16T13:50:30.372 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51305.log 2024-09-16T13:50:30.372 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42916.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104279.log 2024-09-16T13:50:30.372 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.51305.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51305.log.gz 2024-09-16T13:50:30.372 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131117.log 2024-09-16T13:50:30.372 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104279.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30349.log.gz 2024-09-16T13:50:30.372 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122968.log 2024-09-16T13:50:30.373 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57327.log.gz 2024-09-16T13:50:30.373 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.131117.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131117.log.gz 2024-09-16T13:50:30.373 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120249.log 2024-09-16T13:50:30.373 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122968.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.59771.log.gz 2024-09-16T13:50:30.375 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38024.log 2024-09-16T13:50:30.375 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.120249.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27592.log 2024-09-16T13:50:30.375 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53135.log.gz 2024-09-16T13:50:30.375 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.38024.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38024.log.gz 2024-09-16T13:50:30.375 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132896.log 2024-09-16T13:50:30.375 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.27592.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104699.log 2024-09-16T13:50:30.375 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132896.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42223.log.gz 2024-09-16T13:50:30.378 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43126.log 2024-09-16T13:50:30.378 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104699.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90897.log 2024-09-16T13:50:30.378 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43126.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117116.log 2024-09-16T13:50:30.378 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90897.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133044.log 2024-09-16T13:50:30.378 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.117116.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121180.log 2024-09-16T13:50:30.378 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133044.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111368.log 2024-09-16T13:50:30.378 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.121180.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95750.log 2024-09-16T13:50:30.380 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111368.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96256.log 2024-09-16T13:50:30.380 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.95750.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46328.log 2024-09-16T13:50:30.380 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67285.log.gz 2024-09-16T13:50:30.380 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96256.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96256.log.gz 2024-09-16T13:50:30.381 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89552.log 2024-09-16T13:50:30.381 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46328.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122179.log 2024-09-16T13:50:30.383 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89552.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73766.log 2024-09-16T13:50:30.383 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.122179.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.122179.log.gz 2024-09-16T13:50:30.383 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42392.log 2024-09-16T13:50:30.383 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.73766.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78300.log 2024-09-16T13:50:30.384 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42392.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44222.log 2024-09-16T13:50:30.384 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78300.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69476.log.gz 2024-09-16T13:50:30.384 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142013.log 2024-09-16T13:50:30.386 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105165.log 2024-09-16T13:50:30.386 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142013.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51821.log 2024-09-16T13:50:30.386 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105165.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49065.log 2024-09-16T13:50:30.387 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120451.log 2024-09-16T13:50:30.387 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93671.log 2024-09-16T13:50:30.387 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.120451.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134368.log 2024-09-16T13:50:30.389 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93671.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57536.log 2024-09-16T13:50:30.390 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134368.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83769.log 2024-09-16T13:50:30.390 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127341.log 2024-09-16T13:50:30.390 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67919.log.gz 2024-09-16T13:50:30.390 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.83769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123219.log 2024-09-16T13:50:30.390 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113470.log 2024-09-16T13:50:30.390 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123219.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64397.log 2024-09-16T13:50:30.390 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113470.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38408.log 2024-09-16T13:50:30.392 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64397.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31900.log 2024-09-16T13:50:30.392 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.38408.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.95519.log.gz 2024-09-16T13:50:30.393 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63193.log 2024-09-16T13:50:30.393 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31900.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44323.log 2024-09-16T13:50:30.393 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93906.log 2024-09-16T13:50:30.393 INFO:teuthology.orchestra.run.smithi172.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75634.log.gz 2024-09-16T13:50:30.393 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44323.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95820.log 2024-09-16T13:50:30.394 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93906.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113048.log 2024-09-16T13:50:30.394 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103524.log.gz 2024-09-16T13:50:30.394 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.95820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106869.log 2024-09-16T13:50:30.396 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113048.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105229.log.gz 2024-09-16T13:50:30.397 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39752.log 2024-09-16T13:50:30.397 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106869.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129941.log 2024-09-16T13:50:30.397 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81559.log 2024-09-16T13:50:30.397 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.129941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95218.log 2024-09-16T13:50:30.397 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81559.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109990.log 2024-09-16T13:50:30.399 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.95218.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62494.log 2024-09-16T13:50:30.399 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109990.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78300.log.gz 2024-09-16T13:50:30.399 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78317.log.gz 2024-09-16T13:50:30.399 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81959.log 2024-09-16T13:50:30.399 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.62494.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104176.log 2024-09-16T13:50:30.401 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81959.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57849.log 2024-09-16T13:50:30.402 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78464.log.gz 2024-09-16T13:50:30.402 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104176.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.60199.log 2024-09-16T13:50:30.402 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57849.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34964.log 2024-09-16T13:50:30.402 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.60199.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111427.log 2024-09-16T13:50:30.402 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.34964.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34964.log.gz 2024-09-16T13:50:30.402 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28732.log 2024-09-16T13:50:30.402 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88491.log.gz 2024-09-16T13:50:30.402 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111427.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49790.log 2024-09-16T13:50:30.404 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28732.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88573.log.gz 2024-09-16T13:50:30.405 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90432.log 2024-09-16T13:50:30.405 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49790.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49790.log.gz 2024-09-16T13:50:30.405 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110053.log 2024-09-16T13:50:30.405 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90432.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119679.log 2024-09-16T13:50:30.405 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89552.log.gz 2024-09-16T13:50:30.405 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110053.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38548.log 2024-09-16T13:50:30.405 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.119679.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123764.log 2024-09-16T13:50:30.407 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.38548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58756.log 2024-09-16T13:50:30.407 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123764.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52041.log.gz 2024-09-16T13:50:30.407 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53605.log 2024-09-16T13:50:30.407 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62685.log 2024-09-16T13:50:30.407 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53605.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42540.log 2024-09-16T13:50:30.409 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.62685.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143615.log 2024-09-16T13:50:30.409 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42540.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52392.log.gz 2024-09-16T13:50:30.409 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60307.log 2024-09-16T13:50:30.410 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.143615.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90685.log.gz 2024-09-16T13:50:30.410 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78066.log 2024-09-16T13:50:30.410 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.143615.log.gz 2024-09-16T13:50:30.410 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.60307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135693.log 2024-09-16T13:50:30.410 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78066.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61478.log 2024-09-16T13:50:30.411 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135693.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39727.log 2024-09-16T13:50:30.411 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61478.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.61478.log.gz 2024-09-16T13:50:30.411 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43062.log.gz 2024-09-16T13:50:30.411 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110951.log 2024-09-16T13:50:30.413 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39727.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42122.log 2024-09-16T13:50:30.413 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110951.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110421.log 2024-09-16T13:50:30.413 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98268.log 2024-09-16T13:50:30.413 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110421.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45230.log.gz 2024-09-16T13:50:30.415 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30178.log 2024-09-16T13:50:30.415 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98268.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91904.log 2024-09-16T13:50:30.415 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30178.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37991.log 2024-09-16T13:50:30.415 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97916.log 2024-09-16T13:50:30.416 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37991.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37991.log.gz 2024-09-16T13:50:30.416 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85437.log 2024-09-16T13:50:30.416 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.97916.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.97916.log.gz 2024-09-16T13:50:30.416 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.92788.log.gz 2024-09-16T13:50:30.417 INFO:teuthology.orchestra.run.smithi172.stderr:-5 --verbose -- /var/log/ceph/cluster1-client.mirror.66362.log 2024-09-16T13:50:30.418 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102994.log 2024-09-16T13:50:30.418 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122940.log 2024-09-16T13:50:30.418 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102994.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104807.log 2024-09-16T13:50:30.420 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122940.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76376.log 2024-09-16T13:50:30.420 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104807.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.36832.log.gz 2024-09-16T13:50:30.420 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136178.log 2024-09-16T13:50:30.420 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76376.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42225.log 2024-09-16T13:50:30.420 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136178.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64891.log 2024-09-16T13:50:30.422 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42225.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79622.log 2024-09-16T13:50:30.422 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64891.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33835.log 2024-09-16T13:50:30.422 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30980.log.gz 2024-09-16T13:50:30.422 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79622.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56901.log 2024-09-16T13:50:30.422 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33835.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37460.log 2024-09-16T13:50:30.424 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56901.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77351.log 2024-09-16T13:50:30.424 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.37460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89065.log 2024-09-16T13:50:30.425 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28604.log 2024-09-16T13:50:30.425 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89065.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120821.log 2024-09-16T13:50:30.425 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.28604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40693.log 2024-09-16T13:50:30.427 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.120821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73853.log 2024-09-16T13:50:30.427 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90897.log.gz 2024-09-16T13:50:30.427 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.40693.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40693.log.gz 2024-09-16T13:50:30.427 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81457.log 2024-09-16T13:50:30.428 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.73853.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42915.log 2024-09-16T13:50:30.428 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81457.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39215.log 2024-09-16T13:50:30.428 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42915.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141789.log 2024-09-16T13:50:30.429 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39215.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89217.log.gz 2024-09-16T13:50:30.429 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122540.log 2024-09-16T13:50:30.429 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.141789.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66026.log 2024-09-16T13:50:30.430 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122540.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40272.log 2024-09-16T13:50:30.431 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33714.log 2024-09-16T13:50:30.431 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.40272.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40272.log.gz 2024-09-16T13:50:30.431 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82394.log.gz 2024-09-16T13:50:30.431 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43064.log 2024-09-16T13:50:30.432 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33714.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.104257.log.gz 2024-09-16T13:50:30.432 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96906.log 2024-09-16T13:50:30.432 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0%/var/log/ceph/cluster1-client.mirror.43064.log: -- replaced with /var/log/ceph/cluster1-client.mirror.104279.log.gz 2024-09-16T13:50:30.433 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113510.log 2024-09-16T13:50:30.434 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.96906.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137650.log.gz 2024-09-16T13:50:30.434 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64526.log 2024-09-16T13:50:30.434 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67894.log 2024-09-16T13:50:30.435 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64526.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58168.log 2024-09-16T13:50:30.436 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67894.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126874.log.gz 2024-09-16T13:50:30.436 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92493.log 2024-09-16T13:50:30.436 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58168.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107482.log 2024-09-16T13:50:30.436 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92493.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110444.log 2024-09-16T13:50:30.438 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.107482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92789.log 2024-09-16T13:50:30.439 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.110444.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110444.log.gz 2024-09-16T13:50:30.439 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100074.log 2024-09-16T13:50:30.439 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125528.log.gz 2024-09-16T13:50:30.439 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92789.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117806.log 2024-09-16T13:50:30.439 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100074.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.128915.log.gz 2024-09-16T13:50:30.439 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66028.log 2024-09-16T13:50:30.439 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117806.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135589.log 2024-09-16T13:50:30.442 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66028.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26816.log 2024-09-16T13:50:30.442 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135589.log: 87.4%gzip -- replaced with /var/log/ceph/cluster2-client.admin.130303.log.gz-5 2024-09-16T13:50:30.443 INFO:teuthology.orchestra.run.smithi172.stderr:--verbose -- /var/log/ceph/cluster2-client.mirror.71975.log 2024-09-16T13:50:30.443 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.26816.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38361.log 2024-09-16T13:50:30.443 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.71975.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129942.log 2024-09-16T13:50:30.443 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.38361.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105436.log 2024-09-16T13:50:30.443 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.129942.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99241.log 2024-09-16T13:50:30.443 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105436.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46331.log 2024-09-16T13:50:30.443 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99241.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.111427.log.gz 2024-09-16T13:50:30.443 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70037.log 2024-09-16T13:50:30.448 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46331.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50004.log 2024-09-16T13:50:30.448 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.70037.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59988.log 2024-09-16T13:50:30.448 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.50004.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88172.log 2024-09-16T13:50:30.448 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59988.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122968.log.gz 2024-09-16T13:50:30.448 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41762.log 2024-09-16T13:50:30.448 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88172.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126454.log 2024-09-16T13:50:30.448 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41762.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110340.log 2024-09-16T13:50:30.449 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126454.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123765.log.gz 2024-09-16T13:50:30.449 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78404.log 2024-09-16T13:50:30.449 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110340.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131507.log 2024-09-16T13:50:30.449 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78404.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113612.log 2024-09-16T13:50:30.449 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.131507.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62758.log 2024-09-16T13:50:30.449 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113612.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134368.log.gz 2024-09-16T13:50:30.449 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28448.log 2024-09-16T13:50:30.452 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.62758.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139475.log 2024-09-16T13:50:30.452 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.28448.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95523.log 2024-09-16T13:50:30.452 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.120821.log.gz 2024-09-16T13:50:30.452 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.38548.log.gz 2024-09-16T13:50:30.452 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139475.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84026.log 2024-09-16T13:50:30.452 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.95523.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142366.log 2024-09-16T13:50:30.452 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.84026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94270.log 2024-09-16T13:50:30.452 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83770.log 2024-09-16T13:50:30.453 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.94270.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.94270.log.gz 2024-09-16T13:50:30.453 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51821.log.gz 2024-09-16T13:50:30.453 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33924.log 2024-09-16T13:50:30.455 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.83770.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111932.log 2024-09-16T13:50:30.455 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33924.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59773.log 2024-09-16T13:50:30.455 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111932.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-09-16T13:50:30.455 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59773.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53184.log 2024-09-16T13:50:30.455 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87688.log 2024-09-16T13:50:30.455 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53184.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96672.log 2024-09-16T13:50:30.458 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87688.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61717.log 2024-09-16T13:50:30.458 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96672.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96672.log.gz 2024-09-16T13:50:30.458 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30640.log 2024-09-16T13:50:30.458 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.61717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114939.log 2024-09-16T13:50:30.459 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127776.log 2024-09-16T13:50:30.459 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114939.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59769.log 2024-09-16T13:50:30.459 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127776.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110207.log 2024-09-16T13:50:30.462 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41803.log 2024-09-16T13:50:30.462 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110207.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52677.log 2024-09-16T13:50:30.462 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41803.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97316.log 2024-09-16T13:50:30.462 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52677.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130396.log 2024-09-16T13:50:30.462 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.97316.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42183.log 2024-09-16T13:50:30.462 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130396.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.120249.log.gz 2024-09-16T13:50:30.462 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45169.log 2024-09-16T13:50:30.462 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42183.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116245.log 2024-09-16T13:50:30.464 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45169.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115991.log 2024-09-16T13:50:30.464 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116245.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111086.log.gz 2024-09-16T13:50:30.465 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111054.log.gz 2024-09-16T13:50:30.465 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131439.log 2024-09-16T13:50:30.465 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115991.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37386.log 2024-09-16T13:50:30.465 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.131439.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98675.log 2024-09-16T13:50:30.467 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.37386.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99043.log 2024-09-16T13:50:30.467 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98675.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73361.log 2024-09-16T13:50:30.467 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113931.log.gz 2024-09-16T13:50:30.468 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.99043.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98527.log 2024-09-16T13:50:30.468 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73361.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73361.log.gz 2024-09-16T13:50:30.468 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109676.log 2024-09-16T13:50:30.468 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.98527.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.117053.log 2024-09-16T13:50:30.468 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109676.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81352.log 2024-09-16T13:50:30.470 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.117053.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75950.log 2024-09-16T13:50:30.470 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81352.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65820.log 2024-09-16T13:50:30.470 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42392.log.gz 2024-09-16T13:50:30.471 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.75950.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87532.log 2024-09-16T13:50:30.471 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56544.log 2024-09-16T13:50:30.471 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.87532.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120082.log 2024-09-16T13:50:30.471 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56544.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56460.log 2024-09-16T13:50:30.471 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42540.log.gz 2024-09-16T13:50:30.473 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.120082.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127634.log 2024-09-16T13:50:30.473 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88787.log 2024-09-16T13:50:30.473 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127634.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101547.log 2024-09-16T13:50:30.473 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88787.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42544.log 2024-09-16T13:50:30.473 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101547.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46974.log.gz 2024-09-16T13:50:30.473 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55642.log 2024-09-16T13:50:30.476 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42544.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113027.log 2024-09-16T13:50:30.476 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138593.log 2024-09-16T13:50:30.476 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113027.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85403.log 2024-09-16T13:50:30.476 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138593.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82374.log 2024-09-16T13:50:30.476 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85403.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85403.log.gz 2024-09-16T13:50:30.476 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86012.log 2024-09-16T13:50:30.476 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.82374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67079.log 2024-09-16T13:50:30.476 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44222.log.gz 2024-09-16T13:50:30.478 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.86012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131480.log 2024-09-16T13:50:30.478 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67079.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130888.log 2024-09-16T13:50:30.478 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.131480.log: gzip -5 --verbose 89.9%-- /var/log/ceph/cluster1-client.mirror.45862.log -- replaced with /var/log/ceph/cluster1-client.mirror.41762.log.gz 2024-09-16T13:50:30.478 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:30.478 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130888.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130888.log.gz 2024-09-16T13:50:30.478 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50708.log 2024-09-16T13:50:30.480 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45862.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36618.log 2024-09-16T13:50:30.480 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63349.log.gz 2024-09-16T13:50:30.480 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.50708.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126875.log 2024-09-16T13:50:30.480 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36618.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40916.log 2024-09-16T13:50:30.480 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126875.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54591.log 2024-09-16T13:50:30.482 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40916.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72942.log 2024-09-16T13:50:30.483 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54591.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134879.log 2024-09-16T13:50:30.483 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.72942.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133001.log 2024-09-16T13:50:30.483 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134879.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37275.log.gz 2024-09-16T13:50:30.483 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55222.log 2024-09-16T13:50:30.483 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133001.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27812.log 2024-09-16T13:50:30.483 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123427.log 2024-09-16T13:50:30.485 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.27812.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.27812.log.gz 2024-09-16T13:50:30.485 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66658.log 2024-09-16T13:50:30.486 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44854.log.gz 2024-09-16T13:50:30.486 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123427.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.122190.log 2024-09-16T13:50:30.486 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66658.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120879.log 2024-09-16T13:50:30.486 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.122190.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57746.log 2024-09-16T13:50:30.486 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.120879.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44323.log.gz 2024-09-16T13:50:30.486 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127549.log 2024-09-16T13:50:30.488 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57746.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107755.log 2024-09-16T13:50:30.488 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127549.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61466.log 2024-09-16T13:50:30.488 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107755.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88131.log 2024-09-16T13:50:30.488 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61466.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.61466.log.gz 2024-09-16T13:50:30.488 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72831.log 2024-09-16T13:50:30.489 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130377.log 2024-09-16T13:50:30.491 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.72831.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124897.log 2024-09-16T13:50:30.491 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130377.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130377.log.gz 2024-09-16T13:50:30.492 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69183.log 2024-09-16T13:50:30.492 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124897.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108911.log 2024-09-16T13:50:30.492 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46665.log.gz 2024-09-16T13:50:30.492 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69183.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32571.log 2024-09-16T13:50:30.492 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.108911.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50574.log 2024-09-16T13:50:30.492 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32571.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42496.log 2024-09-16T13:50:30.492 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.50574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29338.log 2024-09-16T13:50:30.494 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42496.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29758.log 2024-09-16T13:50:30.495 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29338.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141411.log 2024-09-16T13:50:30.495 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29758.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113728.log 2024-09-16T13:50:30.495 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39727.log.gz 2024-09-16T13:50:30.495 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141411.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86869.log 2024-09-16T13:50:30.495 INFO:teuthology.orchestra.run.smithi172.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.62287.log.gz 2024-09-16T13:50:30.495 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113728.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71271.log 2024-09-16T13:50:30.495 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.86869.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138742.log 2024-09-16T13:50:30.498 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66108.log 2024-09-16T13:50:30.498 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86922.log 2024-09-16T13:50:30.498 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66108.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62152.log 2024-09-16T13:50:30.498 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40498.log.gz 2024-09-16T13:50:30.498 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.86922.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27385.log 2024-09-16T13:50:30.499 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.62152.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32031.log 2024-09-16T13:50:30.499 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.27385.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46221.log 2024-09-16T13:50:30.499 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32031.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.36829.log.gz 2024-09-16T13:50:30.499 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39845.log 2024-09-16T13:50:30.501 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109712.log 2024-09-16T13:50:30.502 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39845.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143086.log 2024-09-16T13:50:30.502 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109712.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110402.log 2024-09-16T13:50:30.502 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.143086.log: 67.5% -- replaced with /var/log/ceph/cluster1-client.mirror.143086.log.gz 2024-09-16T13:50:30.502 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82313.log 2024-09-16T13:50:30.502 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110402.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104114.log 2024-09-16T13:50:30.502 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.82313.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61076.log 2024-09-16T13:50:30.502 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104114.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90475.log 2024-09-16T13:50:30.504 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61076.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80404.log 2024-09-16T13:50:30.505 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90475.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76506.log 2024-09-16T13:50:30.505 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80404.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84379.log 2024-09-16T13:50:30.505 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76506.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96362.log 2024-09-16T13:50:30.505 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.84379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58083.log 2024-09-16T13:50:30.505 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100327.log 2024-09-16T13:50:30.508 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58083.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73926.log 2024-09-16T13:50:30.508 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100327.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112417.log 2024-09-16T13:50:30.508 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.73926.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117225.log 2024-09-16T13:50:30.508 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112417.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135862.log 2024-09-16T13:50:30.508 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117225.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.117225.log.gz 2024-09-16T13:50:30.509 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32660.log 2024-09-16T13:50:30.509 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135862.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28821.log 2024-09-16T13:50:30.509 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32660.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96772.log 2024-09-16T13:50:30.511 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28821.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46181.log 2024-09-16T13:50:30.511 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.96772.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141417.log 2024-09-16T13:50:30.512 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46181.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57535.log 2024-09-16T13:50:30.512 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141417.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45018.log 2024-09-16T13:50:30.512 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57535.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85555.log 2024-09-16T13:50:30.512 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45018.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115084.log 2024-09-16T13:50:30.512 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85555.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85555.log.gz 2024-09-16T13:50:30.512 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131553.log 2024-09-16T13:50:30.514 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115084.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79724.log 2024-09-16T13:50:30.515 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.131553.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142538.log 2024-09-16T13:50:30.515 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79724.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123136.log 2024-09-16T13:50:30.515 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142538.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96710.log 2024-09-16T13:50:30.515 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77834.log 2024-09-16T13:50:30.515 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96710.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96710.log.gz 2024-09-16T13:50:30.515 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56755.log 2024-09-16T13:50:30.518 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77834.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135021.log 2024-09-16T13:50:30.518 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56755.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98992.log 2024-09-16T13:50:30.518 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62629.log 2024-09-16T13:50:30.518 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139775.log 2024-09-16T13:50:30.518 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.62629.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25716.log 2024-09-16T13:50:30.518 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139775.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62956.log 2024-09-16T13:50:30.519 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104807.log.gz 2024-09-16T13:50:30.519 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.25716.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25716.log.gz 2024-09-16T13:50:30.519 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138174.log 2024-09-16T13:50:30.521 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.62956.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112249.log 2024-09-16T13:50:30.521 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138174.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108361.log 2024-09-16T13:50:30.521 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112249.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123301.log 2024-09-16T13:50:30.521 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107482.log.gz 2024-09-16T13:50:30.521 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.108361.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124690.log 2024-09-16T13:50:30.522 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123301.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99281.log 2024-09-16T13:50:30.522 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124690.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123196.log 2024-09-16T13:50:30.524 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99281.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76988.log 2024-09-16T13:50:30.524 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123196.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105436.log.gz 2024-09-16T13:50:30.525 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35077.log 2024-09-16T13:50:30.525 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76988.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99513.log 2024-09-16T13:50:30.525 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.35077.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142523.log 2024-09-16T13:50:30.525 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.86826.log.gz 2024-09-16T13:50:30.525 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.99513.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138912.log 2024-09-16T13:50:30.525 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142523.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142523.log.gz 2024-09-16T13:50:30.525 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112418.log 2024-09-16T13:50:30.528 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138912.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87094.log.gz 2024-09-16T13:50:30.529 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90264.log 2024-09-16T13:50:30.529 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112418.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60713.log 2024-09-16T13:50:30.529 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90264.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98066.log 2024-09-16T13:50:30.529 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.60713.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118838.log 2024-09-16T13:50:30.529 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.98066.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30557.log 2024-09-16T13:50:30.529 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81959.log.gz 2024-09-16T13:50:30.529 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.118838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34994.log 2024-09-16T13:50:30.529 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30557.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75762.log 2024-09-16T13:50:30.529 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34994.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.84213.log.gz 2024-09-16T13:50:30.529 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72860.log 2024-09-16T13:50:30.531 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.75762.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.75762.log.gz 2024-09-16T13:50:30.531 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68047.log 2024-09-16T13:50:30.531 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.72860.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104994.log 2024-09-16T13:50:30.531 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68047.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40978.log 2024-09-16T13:50:30.531 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88787.log.gz 2024-09-16T13:50:30.531 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104994.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32989.log 2024-09-16T13:50:30.533 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40978.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31189.log 2024-09-16T13:50:30.533 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32989.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55641.log 2024-09-16T13:50:30.533 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31189.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110161.log 2024-09-16T13:50:30.533 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.100074.log.gz 2024-09-16T13:50:30.533 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55641.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68468.log 2024-09-16T13:50:30.535 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110161.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32240.log 2024-09-16T13:50:30.535 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68468.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74382.log 2024-09-16T13:50:30.535 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.95750.log.gz 2024-09-16T13:50:30.535 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32240.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133486.log 2024-09-16T13:50:30.536 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.74382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143562.log 2024-09-16T13:50:30.536 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133486.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81352.log.gz 2024-09-16T13:50:30.536 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40445.log 2024-09-16T13:50:30.537 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.143562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46603.log 2024-09-16T13:50:30.537 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.143562.log.gz 2024-09-16T13:50:30.537 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.40445.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40445.log.gz 2024-09-16T13:50:30.538 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82374.log.gz 2024-09-16T13:50:30.538 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50273.log 2024-09-16T13:50:30.539 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46603.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96676.log 2024-09-16T13:50:30.541 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.50273.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68634.log 2024-09-16T13:50:30.541 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96676.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96676.log.gz 2024-09-16T13:50:30.541 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105376.log 2024-09-16T13:50:30.541 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68634.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139970.log 2024-09-16T13:50:30.542 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105376.log: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.93906.log.gz 2024-09-16T13:50:30.542 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125656.log 2024-09-16T13:50:30.542 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.139970.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.139970.log.gz 2024-09-16T13:50:30.543 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34003.log 2024-09-16T13:50:30.543 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125656.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138174.log.gz 2024-09-16T13:50:30.543 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78964.log 2024-09-16T13:50:30.545 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34003.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45651.log 2024-09-16T13:50:30.545 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78964.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51727.log 2024-09-16T13:50:30.546 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45651.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26901.log 2024-09-16T13:50:30.546 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51727.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56693.log 2024-09-16T13:50:30.546 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.26901.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.26901.log.gz 2024-09-16T13:50:30.546 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89275.log 2024-09-16T13:50:30.547 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56693.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31734.log.gz 2024-09-16T13:50:30.548 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134326.log 2024-09-16T13:50:30.549 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89275.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143220.log 2024-09-16T13:50:30.549 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134326.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111015.log 2024-09-16T13:50:30.550 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.143220.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113402.log 2024-09-16T13:50:30.550 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.143220.log.gz 2024-09-16T13:50:30.550 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.111015.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.111015.log.gz 2024-09-16T13:50:30.550 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74309.log 2024-09-16T13:50:30.551 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113402.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88601.log 2024-09-16T13:50:30.551 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.74309.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87779.log 2024-09-16T13:50:30.552 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88601.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67079.log.gz 2024-09-16T13:50:30.552 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71738.log 2024-09-16T13:50:30.553 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87779.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36834.log 2024-09-16T13:50:30.553 INFO:teuthology.orchestra.run.smithi172.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50004.log.gz 2024-09-16T13:50:30.553 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71738.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80378.log 2024-09-16T13:50:30.554 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.36834.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125949.log 2024-09-16T13:50:30.554 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80378.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136911.log 2024-09-16T13:50:30.555 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125949.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142948.log 2024-09-16T13:50:30.556 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136911.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82274.log 2024-09-16T13:50:30.556 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142948.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142948.log.gz 2024-09-16T13:50:30.556 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57218.log 2024-09-16T13:50:30.558 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.82274.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130511.log 2024-09-16T13:50:30.558 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57218.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139861.log.gz 2024-09-16T13:50:30.558 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107479.log 2024-09-16T13:50:30.558 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130511.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129288.log 2024-09-16T13:50:30.558 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.107479.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123134.log 2024-09-16T13:50:30.561 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129288.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33160.log 2024-09-16T13:50:30.561 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135589.log.gz 2024-09-16T13:50:30.561 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123134.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112100.log 2024-09-16T13:50:30.561 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125656.log.gz 2024-09-16T13:50:30.561 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33160.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25743.log 2024-09-16T13:50:30.561 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112100.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74953.log 2024-09-16T13:50:30.561 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.25743.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25743.log.gz 2024-09-16T13:50:30.561 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33291.log 2024-09-16T13:50:30.562 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.74953.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138912.log.gz 2024-09-16T13:50:30.564 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121110.log 2024-09-16T13:50:30.564 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33291.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.131480.log.gz 2024-09-16T13:50:30.564 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30054.log 2024-09-16T13:50:30.564 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121110.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38609.log 2024-09-16T13:50:30.564 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30054.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108516.log 2024-09-16T13:50:30.564 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.131507.log.gz 2024-09-16T13:50:30.565 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.38609.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85138.log 2024-09-16T13:50:30.567 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.108516.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89425.log 2024-09-16T13:50:30.567 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.140882.log 2024-09-16T13:50:30.568 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123974.log.gz 2024-09-16T13:50:30.568 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89425.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42542.log 2024-09-16T13:50:30.568 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.140882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97175.log 2024-09-16T13:50:30.568 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42542.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61052.log 2024-09-16T13:50:30.568 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.97175.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90477.log 2024-09-16T13:50:30.568 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61052.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.61052.log.gz 2024-09-16T13:50:30.568 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43380.log 2024-09-16T13:50:30.570 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90477.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113510.log.gz 2024-09-16T13:50:30.570 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41439.log 2024-09-16T13:50:30.570 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126664.log 2024-09-16T13:50:30.570 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41439.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119010.log 2024-09-16T13:50:30.570 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116245.log.gz 2024-09-16T13:50:30.571 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126664.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56375.log 2024-09-16T13:50:30.572 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.119010.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96265.log 2024-09-16T13:50:30.572 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115780.log 2024-09-16T13:50:30.572 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96265.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96265.log.gz 2024-09-16T13:50:30.572 INFO:teuthology.orchestra.run.smithi172.stderr:gzip 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115084.log.gz-5 2024-09-16T13:50:30.572 INFO:teuthology.orchestra.run.smithi172.stderr:--verbose -- /var/log/ceph/cluster1-client.admin.133126.log 2024-09-16T13:50:30.574 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115780.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37068.log 2024-09-16T13:50:30.575 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.133126.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133126.log.gz 2024-09-16T13:50:30.575 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25617.log 2024-09-16T13:50:30.575 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37068.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.133486.log.gz 2024-09-16T13:50:30.575 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54171.log 2024-09-16T13:50:30.575 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.25617.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25617.log.gz 2024-09-16T13:50:30.575 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144541.log 2024-09-16T13:50:30.578 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54171.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123556.log 2024-09-16T13:50:30.578 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.144541.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144541.log.gz 2024-09-16T13:50:30.578 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83838.log 2024-09-16T13:50:30.578 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123556.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135968.log 2024-09-16T13:50:30.578 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.83838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69100.log 2024-09-16T13:50:30.578 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135968.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106985.log 2024-09-16T13:50:30.578 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69100.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102574.log 2024-09-16T13:50:30.580 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88490.log 2024-09-16T13:50:30.580 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93842.log 2024-09-16T13:50:30.580 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88490.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32991.log 2024-09-16T13:50:30.581 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100401.log 2024-09-16T13:50:30.581 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32991.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141602.log 2024-09-16T13:50:30.583 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100401.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58637.log 2024-09-16T13:50:30.583 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52261.log 2024-09-16T13:50:30.584 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.58637.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.58637.log.gz 2024-09-16T13:50:30.584 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109938.log 2024-09-16T13:50:30.584 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.52261.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91524.log 2024-09-16T13:50:30.584 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.109938.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.109938.log.gz 2024-09-16T13:50:30.584 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102913.log 2024-09-16T13:50:30.584 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91524.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30135.log 2024-09-16T13:50:30.586 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102913.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32661.log 2024-09-16T13:50:30.586 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30135.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101244.log 2024-09-16T13:50:30.586 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32661.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114601.log 2024-09-16T13:50:30.586 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101244.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68764.log 2024-09-16T13:50:30.586 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65814.log 2024-09-16T13:50:30.586 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68764.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139734.log 2024-09-16T13:50:30.588 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65814.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73063.log 2024-09-16T13:50:30.588 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139734.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113728.log.gz 2024-09-16T13:50:30.588 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104868.log 2024-09-16T13:50:30.588 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73063.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73063.log.gz 2024-09-16T13:50:30.589 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116499.log 2024-09-16T13:50:30.589 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104868.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115444.log 2024-09-16T13:50:30.589 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116499.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109541.log 2024-09-16T13:50:30.591 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115444.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97248.log 2024-09-16T13:50:30.591 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62685.log.gz 2024-09-16T13:50:30.591 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91187.log 2024-09-16T13:50:30.591 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.97248.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34994.log.gz 2024-09-16T13:50:30.592 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50092.log 2024-09-16T13:50:30.593 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91187.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70005.log 2024-09-16T13:50:30.593 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.50092.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46073.log 2024-09-16T13:50:30.594 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.70005.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48490.log 2024-09-16T13:50:30.594 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32031.log.gz 2024-09-16T13:50:30.594 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87975.log 2024-09-16T13:50:30.594 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.48490.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110026.log 2024-09-16T13:50:30.596 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87975.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42330.log 2024-09-16T13:50:30.596 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103059.log 2024-09-16T13:50:30.596 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42330.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35077.log.gz 2024-09-16T13:50:30.596 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98565.log 2024-09-16T13:50:30.598 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103059.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64073.log 2024-09-16T13:50:30.598 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98565.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138955.log 2024-09-16T13:50:30.599 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120404.log 2024-09-16T13:50:30.599 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138955.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.141937.log 2024-09-16T13:50:30.599 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29058.log.gz 2024-09-16T13:50:30.601 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.120404.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46010.log 2024-09-16T13:50:30.601 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.141937.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38475.log 2024-09-16T13:50:30.601 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46010.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114662.log 2024-09-16T13:50:30.601 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.38475.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69731.log 2024-09-16T13:50:30.601 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114662.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105481.log 2024-09-16T13:50:30.601 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69731.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39423.log 2024-09-16T13:50:30.602 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105481.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80462.log 2024-09-16T13:50:30.602 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39423.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43064.log.gz 2024-09-16T13:50:30.603 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101019.log 2024-09-16T13:50:30.603 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80462.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115294.log 2024-09-16T13:50:30.605 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29016.log 2024-09-16T13:50:30.605 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115294.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.85804.log 2024-09-16T13:50:30.605 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29016.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109896.log 2024-09-16T13:50:30.607 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.85804.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134497.log 2024-09-16T13:50:30.607 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109896.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108117.log 2024-09-16T13:50:30.607 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134497.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40031.log 2024-09-16T13:50:30.607 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108117.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.108117.log.gz 2024-09-16T13:50:30.607 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87270.log 2024-09-16T13:50:30.609 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40031.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52367.log 2024-09-16T13:50:30.609 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93422.log 2024-09-16T13:50:30.609 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52367.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91103.log 2024-09-16T13:50:30.609 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93422.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79562.log 2024-09-16T13:50:30.611 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29799.log 2024-09-16T13:50:30.611 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74647.log 2024-09-16T13:50:30.611 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78613.log 2024-09-16T13:50:30.611 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.74647.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.144131.log 2024-09-16T13:50:30.612 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78613.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139225.log 2024-09-16T13:50:30.612 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.144131.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.144131.log.gz 2024-09-16T13:50:30.612 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51882.log 2024-09-16T13:50:30.613 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139225.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62665.log 2024-09-16T13:50:30.614 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27928.log 2024-09-16T13:50:30.614 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.62665.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109869.log 2024-09-16T13:50:30.614 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.27928.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91104.log 2024-09-16T13:50:30.614 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109869.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70132.log 2024-09-16T13:50:30.616 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91104.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87530.log 2024-09-16T13:50:30.616 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.70132.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.36830.log 2024-09-16T13:50:30.616 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.87530.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87530.log.gz 2024-09-16T13:50:30.616 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98878.log 2024-09-16T13:50:30.617 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.36830.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95525.log 2024-09-16T13:50:30.617 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.98878.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98878.log.gz 2024-09-16T13:50:30.617 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119512.log 2024-09-16T13:50:30.618 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.95525.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44282.log 2024-09-16T13:50:30.618 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.119512.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48029.log 2024-09-16T13:50:30.618 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44282.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101882.log 2024-09-16T13:50:30.620 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.48029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98419.log 2024-09-16T13:50:30.620 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101882.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76465.log 2024-09-16T13:50:30.620 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98419.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60766.log 2024-09-16T13:50:30.621 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76465.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.111017.log 2024-09-16T13:50:30.621 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.60766.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41655.log 2024-09-16T13:50:30.621 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.111017.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123069.log 2024-09-16T13:50:30.622 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41655.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73564.log 2024-09-16T13:50:30.622 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123069.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107247.log 2024-09-16T13:50:30.624 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.73564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112731.log 2024-09-16T13:50:30.624 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107247.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28416.log 2024-09-16T13:50:30.624 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112731.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131128.log 2024-09-16T13:50:30.625 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28416.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32361.log 2024-09-16T13:50:30.625 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.131128.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131128.log.gz 2024-09-16T13:50:30.625 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32149.log 2024-09-16T13:50:30.626 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32361.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92281.log 2024-09-16T13:50:30.627 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32149.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131781.log 2024-09-16T13:50:30.627 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92281.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42331.log 2024-09-16T13:50:30.630 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.131781.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141870.log 2024-09-16T13:50:30.630 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91441.log 2024-09-16T13:50:30.630 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.141870.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137691.log 2024-09-16T13:50:30.630 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91441.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90977.log 2024-09-16T13:50:30.630 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137691.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131735.log 2024-09-16T13:50:30.630 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90977.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43336.log 2024-09-16T13:50:30.630 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.131735.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64460.log 2024-09-16T13:50:30.633 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43336.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141413.log 2024-09-16T13:50:30.633 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64651.log 2024-09-16T13:50:30.634 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123766.log 2024-09-16T13:50:30.634 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53185.log 2024-09-16T13:50:30.634 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123766.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141414.log 2024-09-16T13:50:30.634 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111890.log 2024-09-16T13:50:30.634 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141414.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.144112.log 2024-09-16T13:50:30.634 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111890.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28057.log 2024-09-16T13:50:30.636 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.144112.log: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.144112.log.gz 2024-09-16T13:50:30.636 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49126.log 2024-09-16T13:50:30.636 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87366.log 2024-09-16T13:50:30.636 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49126.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49126.log.gz 2024-09-16T13:50:30.636 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111563.log 2024-09-16T13:50:30.637 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.87366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67159.log 2024-09-16T13:50:30.638 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111563.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133751.log 2024-09-16T13:50:30.639 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67159.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78362.log 2024-09-16T13:50:30.639 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.133751.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133751.log.gz 2024-09-16T13:50:30.639 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99647.log 2024-09-16T13:50:30.639 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56880.log 2024-09-16T13:50:30.639 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99647.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73678.log 2024-09-16T13:50:30.641 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27677.log 2024-09-16T13:50:30.641 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.73678.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110248.log 2024-09-16T13:50:30.641 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.27677.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101630.log 2024-09-16T13:50:30.641 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.110248.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110248.log.gz 2024-09-16T13:50:30.641 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91860.log 2024-09-16T13:50:30.644 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101630.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55496.log 2024-09-16T13:50:30.644 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91860.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90476.log 2024-09-16T13:50:30.644 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55496.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63931.log 2024-09-16T13:50:30.644 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90476.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34135.log 2024-09-16T13:50:30.645 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63931.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38179.log 2024-09-16T13:50:30.645 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130343.log 2024-09-16T13:50:30.645 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.38179.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109200.log 2024-09-16T13:50:30.646 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130343.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130343.log.gz 2024-09-16T13:50:30.646 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139014.log 2024-09-16T13:50:30.646 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.109200.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80778.log 2024-09-16T13:50:30.647 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113048.log.gz 2024-09-16T13:50:30.647 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139014.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68782.log 2024-09-16T13:50:30.648 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80778.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46603.log.gz 2024-09-16T13:50:30.648 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143159.log 2024-09-16T13:50:30.648 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103836.log 2024-09-16T13:50:30.648 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.143159.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130545.log 2024-09-16T13:50:30.649 INFO:teuthology.orchestra.run.smithi172.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.143159.log.gz 2024-09-16T13:50:30.649 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103836.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93464.log 2024-09-16T13:50:30.650 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124688.log 2024-09-16T13:50:30.650 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128277.log 2024-09-16T13:50:30.651 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124688.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141095.log 2024-09-16T13:50:30.651 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.128277.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97114.log 2024-09-16T13:50:30.652 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141095.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86313.log 2024-09-16T13:50:30.652 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.97114.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124897.log.gz 2024-09-16T13:50:30.653 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31729.log 2024-09-16T13:50:30.654 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.86313.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41025.log 2024-09-16T13:50:30.655 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31729.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132144.log 2024-09-16T13:50:30.655 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41025.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131849.log 2024-09-16T13:50:30.655 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132144.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114140.log 2024-09-16T13:50:30.657 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.131849.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99363.log 2024-09-16T13:50:30.657 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114140.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72552.log 2024-09-16T13:50:30.657 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99363.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40743.log.gz 2024-09-16T13:50:30.657 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88040.log 2024-09-16T13:50:30.657 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.72552.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56964.log 2024-09-16T13:50:30.659 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88040.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136139.log 2024-09-16T13:50:30.659 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56964.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41190.log 2024-09-16T13:50:30.659 INFO:teuthology.orchestra.run.smithi172.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.62629.log.gz 2024-09-16T13:50:30.659 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136139.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34686.log 2024-09-16T13:50:30.659 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41190.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56902.log 2024-09-16T13:50:30.661 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62562.log 2024-09-16T13:50:30.662 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56902.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56401.log 2024-09-16T13:50:30.662 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.62562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129399.log 2024-09-16T13:50:30.662 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56401.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58293.log 2024-09-16T13:50:30.662 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129399.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75941.log 2024-09-16T13:50:30.664 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58293.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110972.log 2024-09-16T13:50:30.664 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50273.log.gz 2024-09-16T13:50:30.664 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.75941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133565.log 2024-09-16T13:50:30.664 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49273.log 2024-09-16T13:50:30.664 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.133565.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30771.log 2024-09-16T13:50:30.667 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49273.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62648.log 2024-09-16T13:50:30.667 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133768.log 2024-09-16T13:50:30.667 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.62648.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38091.log 2024-09-16T13:50:30.667 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.133768.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34633.log 2024-09-16T13:50:30.667 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.38091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65687.log 2024-09-16T13:50:30.667 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34633.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107050.log 2024-09-16T13:50:30.670 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68976.log 2024-09-16T13:50:30.670 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107050.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56126.log 2024-09-16T13:50:30.670 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68976.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60940.log 2024-09-16T13:50:30.670 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56126.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91061.log 2024-09-16T13:50:30.670 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.60940.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82211.log 2024-09-16T13:50:30.671 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91061.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112353.log 2024-09-16T13:50:30.671 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.82211.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.82211.log.gz 2024-09-16T13:50:30.671 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132622.log 2024-09-16T13:50:30.673 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112353.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39958.log 2024-09-16T13:50:30.673 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.132622.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30350.log 2024-09-16T13:50:30.673 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39958.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123099.log 2024-09-16T13:50:30.673 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30350.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114242.log 2024-09-16T13:50:30.673 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68047.log.gz 2024-09-16T13:50:30.673 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123099.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108277.log 2024-09-16T13:50:30.674 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114242.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108026.log 2024-09-16T13:50:30.674 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108277.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.108277.log.gz 2024-09-16T13:50:30.675 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56375.log.gz 2024-09-16T13:50:30.675 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132237.log 2024-09-16T13:50:30.676 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.108026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126162.log 2024-09-16T13:50:30.676 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.132237.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55222.log.gz 2024-09-16T13:50:30.676 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54698.log 2024-09-16T13:50:30.677 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126162.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121248.log 2024-09-16T13:50:30.677 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54698.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.72942.log.gz 2024-09-16T13:50:30.677 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26013.log 2024-09-16T13:50:30.678 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121248.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143200.log 2024-09-16T13:50:30.678 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68468.log.gz 2024-09-16T13:50:30.679 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.26013.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.26013.log.gz 2024-09-16T13:50:30.679 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109139.log 2024-09-16T13:50:30.679 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.143200.log: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.143200.log.gz 2024-09-16T13:50:30.680 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112253.log 2024-09-16T13:50:30.680 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.109139.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63931.log.gz 2024-09-16T13:50:30.680 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135339.log 2024-09-16T13:50:30.681 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112253.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142339.log 2024-09-16T13:50:30.682 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135339.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88415.log 2024-09-16T13:50:30.682 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.142339.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112881.log 2024-09-16T13:50:30.682 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62665.log.gz 2024-09-16T13:50:30.682 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88415.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98700.log 2024-09-16T13:50:30.683 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88332.log 2024-09-16T13:50:30.684 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98700.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48282.log 2024-09-16T13:50:30.684 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65400.log 2024-09-16T13:50:30.685 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.48282.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51173.log 2024-09-16T13:50:30.685 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65400.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88133.log 2024-09-16T13:50:30.687 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.51173.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100536.log 2024-09-16T13:50:30.687 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88133.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59989.log 2024-09-16T13:50:30.687 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34465.log 2024-09-16T13:50:30.688 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59989.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32779.log 2024-09-16T13:50:30.688 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34465.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100181.log 2024-09-16T13:50:30.688 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32779.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135086.log 2024-09-16T13:50:30.692 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100181.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75544.log 2024-09-16T13:50:30.692 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135086.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73678.log.gz 2024-09-16T13:50:30.692 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72056.log 2024-09-16T13:50:30.692 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75544.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142542.log 2024-09-16T13:50:30.692 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.72056.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51045.log 2024-09-16T13:50:30.692 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142542.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26121.log 2024-09-16T13:50:30.692 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.51045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109485.log 2024-09-16T13:50:30.692 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.26121.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.26121.log.gz 2024-09-16T13:50:30.692 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62018.log 2024-09-16T13:50:30.693 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.109485.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37386.log.gz 2024-09-16T13:50:30.693 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.109485.log.gz 2024-09-16T13:50:30.693 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143361.log 2024-09-16T13:50:30.694 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.62018.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81983.log 2024-09-16T13:50:30.695 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.143361.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132314.log 2024-09-16T13:50:30.695 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.37540.log.gz 2024-09-16T13:50:30.695 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81983.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.143361.log.gz 2024-09-16T13:50:30.695 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65399.log 2024-09-16T13:50:30.696 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76159.log 2024-09-16T13:50:30.697 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65399.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67832.log 2024-09-16T13:50:30.697 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.27592.log.gz 2024-09-16T13:50:30.697 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.76159.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.76159.log.gz 2024-09-16T13:50:30.697 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85179.log 2024-09-16T13:50:30.698 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67832.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124664.log 2024-09-16T13:50:30.698 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85179.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98187.log 2024-09-16T13:50:30.698 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124664.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127276.log 2024-09-16T13:50:30.699 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98187.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42122.log.gz 2024-09-16T13:50:30.700 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75773.log 2024-09-16T13:50:30.700 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127276.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29143.log 2024-09-16T13:50:30.701 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.75773.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96253.log 2024-09-16T13:50:30.701 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29143.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72941.log 2024-09-16T13:50:30.701 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96253.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96253.log.gz 2024-09-16T13:50:30.702 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31730.log 2024-09-16T13:50:30.702 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.72941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42014.log 2024-09-16T13:50:30.703 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31730.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.36290.log.gz 2024-09-16T13:50:30.703 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72867.log 2024-09-16T13:50:30.704 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42014.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85991.log 2024-09-16T13:50:30.704 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.72867.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68891.log 2024-09-16T13:50:30.705 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.85991.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67998.log 2024-09-16T13:50:30.705 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68891.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70434.log 2024-09-16T13:50:30.705 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67998.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.73625.log 2024-09-16T13:50:30.706 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9%/var/log/ceph/cluster1-client.admin.70434.log: -- replaced with /var/log/ceph/cluster1-client.mirror.29338.log.gz 2024-09-16T13:50:30.706 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.70434.log.gz 2024-09-16T13:50:30.706 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58462.log 2024-09-16T13:50:30.707 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.73625.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124480.log 2024-09-16T13:50:30.707 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58462.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66867.log 2024-09-16T13:50:30.707 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124480.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128263.log 2024-09-16T13:50:30.708 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66867.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.119699.log 2024-09-16T13:50:30.709 INFO:teuthology.orchestra.run.smithi172.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.109139.log.gz 2024-09-16T13:50:30.709 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.128263.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.128263.log.gz 2024-09-16T13:50:30.709 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79771.log 2024-09-16T13:50:30.709 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.119699.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125084.log 2024-09-16T13:50:30.710 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34958.log 2024-09-16T13:50:30.710 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125084.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84710.log 2024-09-16T13:50:30.711 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.34958.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34958.log.gz 2024-09-16T13:50:30.711 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88601.log.gz 2024-09-16T13:50:30.711 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105119.log 2024-09-16T13:50:30.712 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.84710.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122881.log 2024-09-16T13:50:30.712 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105119.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43905.log 2024-09-16T13:50:30.713 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116290.log 2024-09-16T13:50:30.714 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63678.log 2024-09-16T13:50:30.714 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116290.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.74953.log.gz 2024-09-16T13:50:30.715 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98597.log 2024-09-16T13:50:30.715 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63678.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126286.log 2024-09-16T13:50:30.716 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98597.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.141871.log 2024-09-16T13:50:30.717 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49124.log 2024-09-16T13:50:30.717 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.141871.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95518.log 2024-09-16T13:50:30.717 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49124.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.108026.log.gz 2024-09-16T13:50:30.717 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49124.log.gz 2024-09-16T13:50:30.718 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58227.log 2024-09-16T13:50:30.718 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.95518.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.96018.log.gz 2024-09-16T13:50:30.719 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131761.log 2024-09-16T13:50:30.719 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58227.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62608.log 2024-09-16T13:50:30.720 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.131761.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.107740.log.gz 2024-09-16T13:50:30.720 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43965.log 2024-09-16T13:50:30.720 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.62608.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100746.log 2024-09-16T13:50:30.720 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43965.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57114.log 2024-09-16T13:50:30.722 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103310.log 2024-09-16T13:50:30.722 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57114.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82419.log.gz 2024-09-16T13:50:30.723 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73061.log 2024-09-16T13:50:30.723 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103310.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89928.log 2024-09-16T13:50:30.723 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73061.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73061.log.gz 2024-09-16T13:50:30.723 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90641.log 2024-09-16T13:50:30.723 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88133.log.gz 2024-09-16T13:50:30.724 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89928.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132952.log 2024-09-16T13:50:30.724 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90641.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48304.log 2024-09-16T13:50:30.725 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.132952.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.132952.log.gz 2024-09-16T13:50:30.725 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55435.log 2024-09-16T13:50:30.726 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.48304.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81668.log 2024-09-16T13:50:30.726 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55435.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66824.log 2024-09-16T13:50:30.726 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72860.log.gz 2024-09-16T13:50:30.727 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72823.log.gz 2024-09-16T13:50:30.727 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81668.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44385.log 2024-09-16T13:50:30.729 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66824.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54275.log 2024-09-16T13:50:30.729 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44385.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114981.log 2024-09-16T13:50:30.729 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54275.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90535.log.gz 2024-09-16T13:50:30.729 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90477.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.90476.log.gz 2024-09-16T13:50:30.729 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:30.729 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111723.log.gz 2024-09-16T13:50:30.729 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118310.log 2024-09-16T13:50:30.729 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114981.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114140.log.gz 2024-09-16T13:50:30.732 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75128.log 2024-09-16T13:50:30.732 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118310.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114494.log 2024-09-16T13:50:30.732 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75128.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77566.log 2024-09-16T13:50:30.732 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114494.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.142992.log 2024-09-16T13:50:30.732 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77566.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77771.log 2024-09-16T13:50:30.732 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.142992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36070.log 2024-09-16T13:50:30.732 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.142992.log.gz 2024-09-16T13:50:30.733 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77771.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81518.log 2024-09-16T13:50:30.734 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36070.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138321.log 2024-09-16T13:50:30.735 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81518.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25861.log 2024-09-16T13:50:30.735 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138321.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89508.log 2024-09-16T13:50:30.736 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.25861.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25861.log.gz 2024-09-16T13:50:30.736 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79410.log 2024-09-16T13:50:30.736 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89508.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129404.log 2024-09-16T13:50:30.737 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79410.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52705.log 2024-09-16T13:50:30.738 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129404.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122024.log 2024-09-16T13:50:30.739 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52705.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55011.log 2024-09-16T13:50:30.739 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122024.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39556.log 2024-09-16T13:50:30.739 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55011.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97684.log 2024-09-16T13:50:30.739 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39556.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63398.log 2024-09-16T13:50:30.741 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.97684.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118327.log 2024-09-16T13:50:30.741 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126919.log 2024-09-16T13:50:30.741 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118327.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102785.log 2024-09-16T13:50:30.741 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89760.log 2024-09-16T13:50:30.741 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102785.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52218.log 2024-09-16T13:50:30.743 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89760.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55284.log 2024-09-16T13:50:30.743 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52218.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133994.log 2024-09-16T13:50:30.743 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55284.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78191.log 2024-09-16T13:50:30.743 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.133994.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123404.log 2024-09-16T13:50:30.745 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78191.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26884.log 2024-09-16T13:50:30.745 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123404.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74336.log 2024-09-16T13:50:30.745 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.26884.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75636.log 2024-09-16T13:50:30.745 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.74336.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63038.log 2024-09-16T13:50:30.746 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75636.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34963.log 2024-09-16T13:50:30.746 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63038.log: gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-09-16T13:50:30.747 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.34963.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34963.log.gz 2024-09-16T13:50:30.747 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130886.log 2024-09-16T13:50:30.749 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127695.log 2024-09-16T13:50:30.749 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130886.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130886.log.gz 2024-09-16T13:50:30.749 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30723.log 2024-09-16T13:50:30.750 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127695.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119740.log 2024-09-16T13:50:30.750 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30430.log 2024-09-16T13:50:30.751 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.119740.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118745.log 2024-09-16T13:50:30.751 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30430.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133377.log 2024-09-16T13:50:30.751 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118745.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99624.log 2024-09-16T13:50:30.752 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.133377.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.133377.log.gz 2024-09-16T13:50:30.752 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57389.log 2024-09-16T13:50:30.754 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99624.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108259.log 2024-09-16T13:50:30.754 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67367.log 2024-09-16T13:50:30.754 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108259.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.108259.log.gz 2024-09-16T13:50:30.754 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100072.log 2024-09-16T13:50:30.755 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67367.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138382.log 2024-09-16T13:50:30.755 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100072.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59767.log 2024-09-16T13:50:30.756 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75896.log 2024-09-16T13:50:30.756 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59767.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114873.log 2024-09-16T13:50:30.757 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75896.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96124.log 2024-09-16T13:50:30.758 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114873.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138592.log 2024-09-16T13:50:30.758 INFO:teuthology.orchestra.run.smithi172.stderr: 87.0% -- replaced with /var/log/ceph/cluster2.log.gz 2024-09-16T13:50:30.758 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.96124.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137186.log 2024-09-16T13:50:30.758 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138592.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74268.log 2024-09-16T13:50:30.759 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137186.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98187.log.gz 2024-09-16T13:50:30.759 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102851.log 2024-09-16T13:50:30.759 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.74268.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124539.log 2024-09-16T13:50:30.761 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102851.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48743.log 2024-09-16T13:50:30.761 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124539.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.107922.log 2024-09-16T13:50:30.761 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.48743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39383.log 2024-09-16T13:50:30.761 INFO:teuthology.orchestra.run.smithi172.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.97114.log.gz 2024-09-16T13:50:30.761 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.107922.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105938.log 2024-09-16T13:50:30.763 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.39383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77226.log 2024-09-16T13:50:30.763 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105938.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.21439.log 2024-09-16T13:50:30.764 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57875.log 2024-09-16T13:50:30.764 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.21439.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.21439.log.gz 2024-09-16T13:50:30.764 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47252.log 2024-09-16T13:50:30.765 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57875.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117003.log 2024-09-16T13:50:30.765 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.47252.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76295.log 2024-09-16T13:50:30.765 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117003.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.117003.log.gz 2024-09-16T13:50:30.766 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74020.log 2024-09-16T13:50:30.767 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76295.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116415.log 2024-09-16T13:50:30.767 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.74020.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.114662.log.gz 2024-09-16T13:50:30.767 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114601.log.gz 2024-09-16T13:50:30.768 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85545.log 2024-09-16T13:50:30.768 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116415.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-09-16T13:50:30.768 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85545.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85545.log.gz 2024-09-16T13:50:30.768 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42705.log 2024-09-16T13:50:30.770 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-osd.2.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115444.log.gz 2024-09-16T13:50:30.770 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66844.log 2024-09-16T13:50:30.770 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42705.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31609.log 2024-09-16T13:50:30.770 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.130396.log.gz 2024-09-16T13:50:30.770 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66844.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55852.log 2024-09-16T13:50:30.771 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31609.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34790.log 2024-09-16T13:50:30.771 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55852.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137186.log.gz 2024-09-16T13:50:30.772 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101122.log 2024-09-16T13:50:30.772 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34790.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27353.log 2024-09-16T13:50:30.774 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54720.log 2024-09-16T13:50:30.775 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.27353.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142190.log 2024-09-16T13:50:30.775 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26748.log 2024-09-16T13:50:30.775 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142190.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104048.log 2024-09-16T13:50:30.775 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.136139.log.gz 2024-09-16T13:50:30.775 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.26748.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82024.log 2024-09-16T13:50:30.776 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104048.log: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121248.log.gz 2024-09-16T13:50:30.776 INFO:teuthology.orchestra.run.smithi172.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.121110.log.gz 2024-09-16T13:50:30.776 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53670.log 2024-09-16T13:50:30.778 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.82024.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37602.log 2024-09-16T13:50:30.778 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125084.log.gz 2024-09-16T13:50:30.778 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53670.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68209.log 2024-09-16T13:50:30.779 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135234.log 2024-09-16T13:50:30.779 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68209.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29101.log 2024-09-16T13:50:30.779 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135234.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89131.log 2024-09-16T13:50:30.780 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99432.log 2024-09-16T13:50:30.781 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131095.log 2024-09-16T13:50:30.781 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99432.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41254.log 2024-09-16T13:50:30.781 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.131095.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131095.log.gz 2024-09-16T13:50:30.781 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72386.log 2024-09-16T13:50:30.783 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41254.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.47895.log 2024-09-16T13:50:30.783 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.72386.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90642.log 2024-09-16T13:50:30.783 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.47895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67368.log 2024-09-16T13:50:30.783 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102283.log 2024-09-16T13:50:30.785 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67368.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91062.log 2024-09-16T13:50:30.785 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76842.log 2024-09-16T13:50:30.785 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114517.log 2024-09-16T13:50:30.785 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76842.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92957.log 2024-09-16T13:50:30.787 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114517.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.144181.log 2024-09-16T13:50:30.787 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92957.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63639.log 2024-09-16T13:50:30.787 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126162.log.gz 2024-09-16T13:50:30.787 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.144181.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.144181.log.gz 2024-09-16T13:50:30.787 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120155.log 2024-09-16T13:50:30.789 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.63639.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29673.log 2024-09-16T13:50:30.789 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.120155.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69896.log 2024-09-16T13:50:30.789 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29673.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139734.log.gz 2024-09-16T13:50:30.789 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.82270.log 2024-09-16T13:50:30.790 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69896.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55705.log 2024-09-16T13:50:30.790 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.82270.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.82270.log.gz 2024-09-16T13:50:30.790 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139122.log 2024-09-16T13:50:30.792 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55705.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45701.log 2024-09-16T13:50:30.792 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139122.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91188.log 2024-09-16T13:50:30.792 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45701.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89970.log 2024-09-16T13:50:30.793 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91188.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125526.log 2024-09-16T13:50:30.793 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.103836.log.gz 2024-09-16T13:50:30.793 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89970.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77100.log 2024-09-16T13:50:30.794 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125526.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117641.log 2024-09-16T13:50:30.794 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77100.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86961.log 2024-09-16T13:50:30.795 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117641.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29226.log 2024-09-16T13:50:30.795 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.86961.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89275.log.gz 2024-09-16T13:50:30.795 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125740.log 2024-09-16T13:50:30.796 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29226.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28126.log 2024-09-16T13:50:30.797 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125740.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136347.log 2024-09-16T13:50:30.797 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.28126.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97544.log 2024-09-16T13:50:30.797 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136347.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108817.log 2024-09-16T13:50:30.798 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.97544.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27091.log 2024-09-16T13:50:30.799 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.108817.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53250.log 2024-09-16T13:50:30.799 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125820.log.gz 2024-09-16T13:50:30.799 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.27091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79874.log 2024-09-16T13:50:30.800 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53250.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108657.log 2024-09-16T13:50:30.800 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79874.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29144.log 2024-09-16T13:50:30.800 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.108657.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73193.log 2024-09-16T13:50:30.800 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29144.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45800.log 2024-09-16T13:50:30.802 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73193.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107524.log 2024-09-16T13:50:30.802 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.99281.log.gz 2024-09-16T13:50:30.802 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45800.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72674.log 2024-09-16T13:50:30.802 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107524.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.107524.log.gz 2024-09-16T13:50:30.803 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109045.log 2024-09-16T13:50:30.803 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.72674.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82274.log.gz 2024-09-16T13:50:30.804 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141606.log 2024-09-16T13:50:30.804 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.109045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112562.log 2024-09-16T13:50:30.805 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141606.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86629.log 2024-09-16T13:50:30.806 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48116.log 2024-09-16T13:50:30.806 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.86629.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85898.log 2024-09-16T13:50:30.806 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.48116.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56062.log 2024-09-16T13:50:30.809 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.85898.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87426.log 2024-09-16T13:50:30.810 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41439.log.gz 2024-09-16T13:50:30.810 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136138.log 2024-09-16T13:50:30.810 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135652.log 2024-09-16T13:50:30.810 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108496.log 2024-09-16T13:50:30.810 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102430.log 2024-09-16T13:50:30.810 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.108496.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73062.log 2024-09-16T13:50:30.810 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102430.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.110026.log.gz 2024-09-16T13:50:30.812 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44431.log 2024-09-16T13:50:30.812 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73062.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73062.log.gz 2024-09-16T13:50:30.812 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.75941.log.gz 2024-09-16T13:50:30.812 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45549.log 2024-09-16T13:50:30.812 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82222.log 2024-09-16T13:50:30.813 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45549.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107741.log 2024-09-16T13:50:30.814 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.82222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.144018.log 2024-09-16T13:50:30.814 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89718.log 2024-09-16T13:50:30.814 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.144018.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.144018.log.gz 2024-09-16T13:50:30.814 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.95428.log 2024-09-16T13:50:30.816 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89718.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34973.log 2024-09-16T13:50:30.816 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.95428.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141590.log 2024-09-16T13:50:30.818 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34973.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103416.log 2024-09-16T13:50:30.818 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141590.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110561.log 2024-09-16T13:50:30.818 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103416.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115550.log 2024-09-16T13:50:30.818 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110561.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53469.log 2024-09-16T13:50:30.821 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115550.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121286.log 2024-09-16T13:50:30.821 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53469.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143428.log 2024-09-16T13:50:30.821 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121286.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32989.log.gz 2024-09-16T13:50:30.821 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91945.log 2024-09-16T13:50:30.821 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.143428.log: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.143428.log.gz 2024-09-16T13:50:30.821 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32991.log.gz 2024-09-16T13:50:30.821 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54594.log 2024-09-16T13:50:30.822 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33792.log 2024-09-16T13:50:30.823 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54594.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48743.log.gz 2024-09-16T13:50:30.823 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109674.log 2024-09-16T13:50:30.824 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33792.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.50092.log.gz 2024-09-16T13:50:30.824 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48883.log.gz 2024-09-16T13:50:30.824 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92494.log 2024-09-16T13:50:30.824 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109674.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69054.log 2024-09-16T13:50:30.825 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92494.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52218.log.gz 2024-09-16T13:50:30.825 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74175.log 2024-09-16T13:50:30.827 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69054.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.27146.log 2024-09-16T13:50:30.827 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90977.log.gz 2024-09-16T13:50:30.827 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.74175.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94403.log 2024-09-16T13:50:30.827 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.0.27146.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32529.log 2024-09-16T13:50:30.828 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.94403.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.94403.log.gz 2024-09-16T13:50:30.828 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128245.log 2024-09-16T13:50:30.829 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32529.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52834.log 2024-09-16T13:50:30.829 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.128245.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.128245.log.gz 2024-09-16T13:50:30.829 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120135.log 2024-09-16T13:50:30.829 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52834.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64527.log 2024-09-16T13:50:30.831 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.120135.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43801.log 2024-09-16T13:50:30.831 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64527.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66528.log 2024-09-16T13:50:30.831 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43801.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82067.log 2024-09-16T13:50:30.831 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66528.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100471.log 2024-09-16T13:50:30.833 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.82067.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137399.log 2024-09-16T13:50:30.833 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100471.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73388.log 2024-09-16T13:50:30.833 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137399.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134242.log 2024-09-16T13:50:30.833 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73388.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73388.log.gz 2024-09-16T13:50:30.833 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36477.log 2024-09-16T13:50:30.834 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134242.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139373.log 2024-09-16T13:50:30.835 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74536.log 2024-09-16T13:50:30.835 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139373.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115360.log 2024-09-16T13:50:30.835 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.74536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138659.log 2024-09-16T13:50:30.836 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115360.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28135.log 2024-09-16T13:50:30.837 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138659.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29382.log 2024-09-16T13:50:30.837 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.28135.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84025.log 2024-09-16T13:50:30.838 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116246.log 2024-09-16T13:50:30.838 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.84025.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97818.log 2024-09-16T13:50:30.838 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116246.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67834.log 2024-09-16T13:50:30.839 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.97818.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79770.log 2024-09-16T13:50:30.839 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67834.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37608.log 2024-09-16T13:50:30.840 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79770.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122772.log 2024-09-16T13:50:30.840 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37608.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87817.log 2024-09-16T13:50:30.841 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122772.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85743.log 2024-09-16T13:50:30.842 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87817.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75087.log 2024-09-16T13:50:30.842 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.85743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36478.log 2024-09-16T13:50:30.843 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75087.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89844.log 2024-09-16T13:50:30.843 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36478.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108264.log 2024-09-16T13:50:30.843 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89844.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119350.log 2024-09-16T13:50:30.844 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108264.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.108264.log.gz 2024-09-16T13:50:30.845 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37062.log 2024-09-16T13:50:30.845 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.119350.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119350.log.gz 2024-09-16T13:50:30.845 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97590.log 2024-09-16T13:50:30.846 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42961.log 2024-09-16T13:50:30.846 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.97590.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112101.log 2024-09-16T13:50:30.846 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42961.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39993.log 2024-09-16T13:50:30.847 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43966.log 2024-09-16T13:50:30.847 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.39993.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128055.log 2024-09-16T13:50:30.848 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43966.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37353.log 2024-09-16T13:50:30.848 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.128055.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48705.log 2024-09-16T13:50:30.849 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.37353.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58636.log 2024-09-16T13:50:30.849 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.48705.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39238.log 2024-09-16T13:50:30.849 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.58636.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.58636.log.gz 2024-09-16T13:50:30.850 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119653.log 2024-09-16T13:50:30.850 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.39238.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118963.log 2024-09-16T13:50:30.851 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.119653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67917.log 2024-09-16T13:50:30.852 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118963.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88534.log 2024-09-16T13:50:30.852 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67917.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114937.log 2024-09-16T13:50:30.852 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88534.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89215.log 2024-09-16T13:50:30.853 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114937.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68347.log 2024-09-16T13:50:30.854 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36068.log 2024-09-16T13:50:30.854 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68347.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54360.log 2024-09-16T13:50:30.854 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36068.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44072.log 2024-09-16T13:50:30.856 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54360.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81768.log 2024-09-16T13:50:30.856 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44072.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92915.log 2024-09-16T13:50:30.856 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81768.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26765.log 2024-09-16T13:50:30.857 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92915.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45069.log 2024-09-16T13:50:30.857 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.26765.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62106.log 2024-09-16T13:50:30.857 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45069.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102806.log 2024-09-16T13:50:30.859 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.62106.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39640.log 2024-09-16T13:50:30.859 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102806.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49014.log 2024-09-16T13:50:30.859 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.39640.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39640.log.gz 2024-09-16T13:50:30.859 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42603.log 2024-09-16T13:50:30.860 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49014.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101525.log 2024-09-16T13:50:30.860 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42603.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74978.log 2024-09-16T13:50:30.860 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101525.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57303.log 2024-09-16T13:50:30.861 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.74978.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44915.log 2024-09-16T13:50:30.861 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125236.log 2024-09-16T13:50:30.862 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44915.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.135862.log.gz 2024-09-16T13:50:30.862 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144674.log 2024-09-16T13:50:30.863 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125236.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55768.log 2024-09-16T13:50:30.864 INFO:teuthology.orchestra.run.smithi172.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.131761.log.gz 2024-09-16T13:50:30.864 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.144674.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144674.log.gz 2024-09-16T13:50:30.864 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44640.log 2024-09-16T13:50:30.865 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55768.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138109.log 2024-09-16T13:50:30.865 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44640.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135169.log 2024-09-16T13:50:30.865 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138109.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118636.log 2024-09-16T13:50:30.867 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135169.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140170.log 2024-09-16T13:50:30.867 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118636.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92368.log 2024-09-16T13:50:30.867 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.140170.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74248.log 2024-09-16T13:50:30.868 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92368.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55852.log.gz 2024-09-16T13:50:30.868 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33712.log 2024-09-16T13:50:30.869 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.74248.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71274.log 2024-09-16T13:50:30.869 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33712.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123555.log 2024-09-16T13:50:30.870 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57875.log.gz 2024-09-16T13:50:30.870 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71274.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97858.log 2024-09-16T13:50:30.871 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123555.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111604.log 2024-09-16T13:50:30.871 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.97858.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101314.log 2024-09-16T13:50:30.871 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110376.log 2024-09-16T13:50:30.872 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75247.log 2024-09-16T13:50:30.873 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110376.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42643.log 2024-09-16T13:50:30.873 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75247.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72045.log 2024-09-16T13:50:30.874 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42643.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84570.log 2024-09-16T13:50:30.874 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.72045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33199.log 2024-09-16T13:50:30.874 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.84570.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70437.log 2024-09-16T13:50:30.875 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33199.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30933.log 2024-09-16T13:50:30.875 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.70437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128075.log 2024-09-16T13:50:30.876 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30933.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93002.log 2024-09-16T13:50:30.876 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.128075.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125108.log 2024-09-16T13:50:30.877 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93002.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49712.log 2024-09-16T13:50:30.878 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125108.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34136.log 2024-09-16T13:50:30.878 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49712.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46856.log 2024-09-16T13:50:30.879 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34136.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133709.log 2024-09-16T13:50:30.879 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.46856.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46856.log.gz 2024-09-16T13:50:30.879 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66992.log 2024-09-16T13:50:30.880 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133709.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79243.log 2024-09-16T13:50:30.880 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66992.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49750.log 2024-09-16T13:50:30.881 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79243.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124244.log 2024-09-16T13:50:30.881 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49750.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68256.log 2024-09-16T13:50:30.882 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124244.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142011.log 2024-09-16T13:50:30.882 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68256.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50936.log 2024-09-16T13:50:30.883 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142011.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26200.log 2024-09-16T13:50:30.883 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.50936.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90182.log 2024-09-16T13:50:30.884 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.26200.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.26200.log.gz 2024-09-16T13:50:30.884 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114139.log 2024-09-16T13:50:30.885 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90182.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60514.log 2024-09-16T13:50:30.885 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114139.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74999.log 2024-09-16T13:50:30.885 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.60514.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36483.log 2024-09-16T13:50:30.886 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.74999.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56585.log 2024-09-16T13:50:30.887 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36483.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134286.log 2024-09-16T13:50:30.887 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56585.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88880.log 2024-09-16T13:50:30.887 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134286.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132526.log 2024-09-16T13:50:30.888 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88880.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54275.log.gz 2024-09-16T13:50:30.888 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115990.log 2024-09-16T13:50:30.889 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132526.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142644.log 2024-09-16T13:50:30.889 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115990.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66844.log.gz 2024-09-16T13:50:30.890 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81729.log 2024-09-16T13:50:30.891 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77897.log 2024-09-16T13:50:30.892 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81729.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51575.log 2024-09-16T13:50:30.892 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77897.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125317.log 2024-09-16T13:50:30.892 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51575.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85168.log 2024-09-16T13:50:30.894 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125317.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85784.log 2024-09-16T13:50:30.894 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54594.log.gz 2024-09-16T13:50:30.894 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44807.log 2024-09-16T13:50:30.894 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.85784.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60918.log 2024-09-16T13:50:30.894 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44807.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53469.log.gz 2024-09-16T13:50:30.895 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132653.log 2024-09-16T13:50:30.896 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.60918.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96294.log 2024-09-16T13:50:30.896 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132653.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131346.log 2024-09-16T13:50:30.897 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96294.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67998.log.gz 2024-09-16T13:50:30.897 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128402.log 2024-09-16T13:50:30.898 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.131346.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136389.log 2024-09-16T13:50:30.898 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.128402.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.128402.log.gz 2024-09-16T13:50:30.898 INFO:teuthology.orchestra.run.smithi172.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62018.log.gz 2024-09-16T13:50:30.899 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92071.log 2024-09-16T13:50:30.900 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136389.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126793.log 2024-09-16T13:50:30.900 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92071.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70174.log 2024-09-16T13:50:30.901 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122095.log 2024-09-16T13:50:30.902 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.70174.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.74020.log.gz 2024-09-16T13:50:30.902 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63628.log 2024-09-16T13:50:30.902 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122095.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57600.log 2024-09-16T13:50:30.902 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.63628.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63628.log.gz 2024-09-16T13:50:30.903 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142733.log 2024-09-16T13:50:30.903 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57600.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79202.log 2024-09-16T13:50:30.903 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142733.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86333.log 2024-09-16T13:50:30.904 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79202.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80778.log.gz 2024-09-16T13:50:30.905 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132590.log 2024-09-16T13:50:30.905 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.86333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57032.log 2024-09-16T13:50:30.906 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132590.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104808.log 2024-09-16T13:50:30.906 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57032.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27727.log 2024-09-16T13:50:30.906 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104808.log: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.39423.log.gz 2024-09-16T13:50:30.906 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71269.log 2024-09-16T13:50:30.907 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.27727.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62988.log 2024-09-16T13:50:30.907 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71269.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73812.log 2024-09-16T13:50:30.908 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.62988.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37069.log 2024-09-16T13:50:30.909 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.73812.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61650.log 2024-09-16T13:50:30.909 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37069.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62845.log 2024-09-16T13:50:30.909 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.61650.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142946.log 2024-09-16T13:50:30.910 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.62845.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137121.log 2024-09-16T13:50:30.910 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100746.log.gz 2024-09-16T13:50:30.910 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142946.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142946.log.gz 2024-09-16T13:50:30.911 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119204.log 2024-09-16T13:50:30.912 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137121.log: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.85084.log.gz 2024-09-16T13:50:30.912 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49091.log 2024-09-16T13:50:30.912 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.119204.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141100.log 2024-09-16T13:50:30.913 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.49091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78091.log 2024-09-16T13:50:30.913 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141100.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84287.log 2024-09-16T13:50:30.913 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31309.log 2024-09-16T13:50:30.914 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.84287.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142010.log 2024-09-16T13:50:30.914 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31309.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88742.log 2024-09-16T13:50:30.914 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88880.log.gz 2024-09-16T13:50:30.915 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142010.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90057.log 2024-09-16T13:50:30.915 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91188.log.gz 2024-09-16T13:50:30.916 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88742.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141249.log 2024-09-16T13:50:30.916 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90057.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96480.log 2024-09-16T13:50:30.917 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141249.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81768.log.gz 2024-09-16T13:50:30.917 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68472.log 2024-09-16T13:50:30.918 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96480.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114602.log 2024-09-16T13:50:30.918 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68472.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84223.log 2024-09-16T13:50:30.919 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64071.log 2024-09-16T13:50:30.919 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.84223.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81983.log.gz 2024-09-16T13:50:30.919 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130901.log 2024-09-16T13:50:30.920 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64071.log: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.95518.log.gz 2024-09-16T13:50:30.920 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.82067.log.gz 2024-09-16T13:50:30.921 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.82024.log.gz 2024-09-16T13:50:30.921 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33201.log 2024-09-16T13:50:30.922 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130901.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44430.log 2024-09-16T13:50:30.922 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33201.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135591.log 2024-09-16T13:50:30.923 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44430.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101422.log 2024-09-16T13:50:30.923 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135591.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46909.log 2024-09-16T13:50:30.924 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101422.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43174.log 2024-09-16T13:50:30.925 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46909.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27057.log 2024-09-16T13:50:30.925 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43174.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60505.log 2024-09-16T13:50:30.926 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.27057.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63540.log 2024-09-16T13:50:30.926 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.60505.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53003.log 2024-09-16T13:50:30.927 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63540.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107637.log 2024-09-16T13:50:30.928 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.85898.log.gz 2024-09-16T13:50:30.928 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53003.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96129.log 2024-09-16T13:50:30.928 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107637.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65315.log 2024-09-16T13:50:30.928 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.96129.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37564.log 2024-09-16T13:50:30.929 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65315.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.70321.log 2024-09-16T13:50:30.930 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105228.log 2024-09-16T13:50:30.930 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.70321.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92957.log.gz 2024-09-16T13:50:30.930 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.70321.log.gz 2024-09-16T13:50:30.930 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94159.log 2024-09-16T13:50:30.931 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89592.log 2024-09-16T13:50:30.932 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.94159.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46794.log 2024-09-16T13:50:30.932 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89592.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90955.log 2024-09-16T13:50:30.932 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.46794.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46794.log.gz 2024-09-16T13:50:30.932 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37514.log 2024-09-16T13:50:30.933 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90955.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114602.log.gz 2024-09-16T13:50:30.933 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109806.log 2024-09-16T13:50:30.933 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37514.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.37514.log.gz 2024-09-16T13:50:30.934 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105040.log 2024-09-16T13:50:30.935 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109806.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117640.log 2024-09-16T13:50:30.935 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105040.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91859.log 2024-09-16T13:50:30.935 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.108496.log.gz 2024-09-16T13:50:30.935 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117640.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118844.log 2024-09-16T13:50:30.937 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91859.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41552.log 2024-09-16T13:50:30.937 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.118844.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66573.log 2024-09-16T13:50:30.937 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41552.log: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.96294.log.gz 2024-09-16T13:50:30.937 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61228.log 2024-09-16T13:50:30.938 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66573.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52858.log 2024-09-16T13:50:30.938 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53026.log 2024-09-16T13:50:30.939 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52858.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28525.log 2024-09-16T13:50:30.939 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103692.log 2024-09-16T13:50:30.940 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28525.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.131460.log 2024-09-16T13:50:30.940 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63893.log 2024-09-16T13:50:30.941 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.131460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110074.log 2024-09-16T13:50:30.941 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63893.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.109674.log.gz 2024-09-16T13:50:30.941 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137373.log 2024-09-16T13:50:30.941 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.110074.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.110074.log.gz 2024-09-16T13:50:30.942 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83840.log 2024-09-16T13:50:30.942 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137373.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58645.log 2024-09-16T13:50:30.943 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.83840.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65103.log 2024-09-16T13:50:30.943 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.58645.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.58645.log.gz 2024-09-16T13:50:30.943 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41802.log 2024-09-16T13:50:30.944 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65103.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136389.log.gz 2024-09-16T13:50:30.944 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115295.log 2024-09-16T13:50:30.945 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41802.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34134.log 2024-09-16T13:50:30.945 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115295.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36676.log 2024-09-16T13:50:30.946 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34134.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100182.log 2024-09-16T13:50:30.947 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36676.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115401.log 2024-09-16T13:50:30.947 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100182.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39984.log 2024-09-16T13:50:30.947 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115401.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135441.log 2024-09-16T13:50:30.948 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.39984.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46220.log 2024-09-16T13:50:30.948 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135441.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136073.log 2024-09-16T13:50:30.949 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46220.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122317.log 2024-09-16T13:50:30.950 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123554.log 2024-09-16T13:50:30.950 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122317.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108972.log 2024-09-16T13:50:30.950 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125317.log.gz 2024-09-16T13:50:30.951 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123554.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116878.log 2024-09-16T13:50:30.951 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.108972.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68783.log 2024-09-16T13:50:30.952 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116878.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108240.log 2024-09-16T13:50:30.952 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68783.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98929.log 2024-09-16T13:50:30.952 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108240.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.108240.log.gz 2024-09-16T13:50:30.954 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142524.log 2024-09-16T13:50:30.954 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61330.log 2024-09-16T13:50:30.954 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142524.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142524.log.gz 2024-09-16T13:50:30.954 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88786.log 2024-09-16T13:50:30.955 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61330.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43022.log 2024-09-16T13:50:30.956 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88786.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53670.log.gz 2024-09-16T13:50:30.956 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63959.log 2024-09-16T13:50:30.957 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43022.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99772.log 2024-09-16T13:50:30.957 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63959.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92579.log 2024-09-16T13:50:30.958 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99772.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141099.log 2024-09-16T13:50:30.958 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92579.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115990.log.gz 2024-09-16T13:50:30.959 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32030.log 2024-09-16T13:50:30.960 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141099.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113257.log 2024-09-16T13:50:30.960 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32030.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48548.log 2024-09-16T13:50:30.962 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113257.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131279.log 2024-09-16T13:50:30.962 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.48548.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61066.log 2024-09-16T13:50:30.962 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.131279.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94937.log 2024-09-16T13:50:30.962 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34003.log.gz 2024-09-16T13:50:30.962 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61066.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73947.log 2024-09-16T13:50:30.963 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.94937.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105333.log 2024-09-16T13:50:30.963 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.73947.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125799.log 2024-09-16T13:50:30.964 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.39215.log.gz 2024-09-16T13:50:30.964 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66445.log 2024-09-16T13:50:30.972 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125799.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37252.log 2024-09-16T13:50:30.973 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66445.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98880.log 2024-09-16T13:50:30.973 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.37252.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49138.log 2024-09-16T13:50:30.973 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.98880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44871.log 2024-09-16T13:50:30.973 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101336.log 2024-09-16T13:50:30.973 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44871.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101235.log 2024-09-16T13:50:30.973 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101336.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79537.log 2024-09-16T13:50:30.973 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101235.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64331.log 2024-09-16T13:50:30.973 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79537.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.96086.log 2024-09-16T13:50:30.973 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.64331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62706.log 2024-09-16T13:50:30.973 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.96086.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92368.log.gz 2024-09-16T13:50:30.973 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56250.log 2024-09-16T13:50:30.973 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.62706.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62706.log.gz 2024-09-16T13:50:30.974 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131191.log 2024-09-16T13:50:30.974 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56250.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92494.log.gz 2024-09-16T13:50:30.974 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49120.log 2024-09-16T13:50:30.974 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.131191.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124329.log 2024-09-16T13:50:30.974 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.37564.log.gz 2024-09-16T13:50:30.974 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49120.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49120.log.gz 2024-09-16T13:50:30.974 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.29144.log.gz 2024-09-16T13:50:30.974 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126708.log 2024-09-16T13:50:30.974 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124329.log: 17.4% -- replaced with /var/log/ceph/cluster1-client.admin.39238.log.gz 2024-09-16T13:50:30.976 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77672.log 2024-09-16T13:50:30.976 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126708.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56608.log 2024-09-16T13:50:30.977 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77672.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80043.log 2024-09-16T13:50:30.977 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.28212.log.gz 2024-09-16T13:50:30.977 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56608.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114811.log 2024-09-16T13:50:30.977 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80043.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31820.log 2024-09-16T13:50:30.977 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114811.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65734.log 2024-09-16T13:50:30.977 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31820.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134222.log 2024-09-16T13:50:30.978 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65734.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62796.log 2024-09-16T13:50:30.978 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34344.log 2024-09-16T13:50:30.978 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.62796.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91565.log 2024-09-16T13:50:30.979 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34344.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37230.log 2024-09-16T13:50:30.979 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91565.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110305.log 2024-09-16T13:50:30.980 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.37230.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38750.log 2024-09-16T13:50:30.980 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110305.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67833.log 2024-09-16T13:50:30.981 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.38750.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78992.log 2024-09-16T13:50:30.981 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67833.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49197.log 2024-09-16T13:50:30.982 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78992.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.21261.log 2024-09-16T13:50:30.982 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49197.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61491.log 2024-09-16T13:50:30.983 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.21261.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.21261.log.gz 2024-09-16T13:50:30.983 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57452.log 2024-09-16T13:50:30.983 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61491.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.61491.log.gz 2024-09-16T13:50:30.983 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41594.log 2024-09-16T13:50:30.984 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57452.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120647.log 2024-09-16T13:50:30.985 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41594.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27449.log 2024-09-16T13:50:30.986 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.120647.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118088.log 2024-09-16T13:50:30.986 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.27449.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57179.log 2024-09-16T13:50:30.986 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69054.log.gz 2024-09-16T13:50:30.986 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.118088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110909.log 2024-09-16T13:50:30.987 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57179.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93967.log 2024-09-16T13:50:30.987 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110909.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135507.log 2024-09-16T13:50:30.988 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93967.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27485.log 2024-09-16T13:50:30.989 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135507.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81309.log 2024-09-16T13:50:30.989 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.27485.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42014.log.gz 2024-09-16T13:50:30.989 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42813.log 2024-09-16T13:50:30.990 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81309.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70438.log 2024-09-16T13:50:30.990 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42813.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127005.log 2024-09-16T13:50:30.991 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.70438.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75429.log 2024-09-16T13:50:30.991 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127005.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34423.log.gz 2024-09-16T13:50:30.991 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44492.log 2024-09-16T13:50:30.992 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75429.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45231.log 2024-09-16T13:50:30.992 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44492.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58643.log 2024-09-16T13:50:30.993 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45231.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112982.log 2024-09-16T13:50:30.993 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.58643.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.58643.log.gz 2024-09-16T13:50:30.993 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138324.log 2024-09-16T13:50:30.994 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112982.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97061.log 2024-09-16T13:50:30.995 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138324.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36072.log 2024-09-16T13:50:30.996 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.97061.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55435.log.gz 2024-09-16T13:50:30.996 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68210.log 2024-09-16T13:50:30.997 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36072.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140004.log 2024-09-16T13:50:30.997 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68210.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115336.log 2024-09-16T13:50:30.997 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.140004.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123510.log 2024-09-16T13:50:30.998 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115336.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38038.log 2024-09-16T13:50:30.999 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123510.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70525.log 2024-09-16T13:50:30.999 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.38038.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85337.log 2024-09-16T13:50:31.000 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.70525.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61056.log 2024-09-16T13:50:31.000 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.85337.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.129940.log 2024-09-16T13:50:31.001 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61056.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.61056.log.gz 2024-09-16T13:50:31.001 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.139997.log 2024-09-16T13:50:31.002 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.129940.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38132.log 2024-09-16T13:50:31.002 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.139997.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.139997.log.gz 2024-09-16T13:50:31.002 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109159.log 2024-09-16T13:50:31.003 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.38132.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130426.log 2024-09-16T13:50:31.004 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.109159.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74222.log 2024-09-16T13:50:31.004 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130426.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66026.log.gz 2024-09-16T13:50:31.004 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68044.log 2024-09-16T13:50:31.004 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.74222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55223.log 2024-09-16T13:50:31.005 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68044.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108543.log 2024-09-16T13:50:31.005 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55223.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.38011.log 2024-09-16T13:50:31.006 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.108543.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136767.log 2024-09-16T13:50:31.006 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.38011.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.38011.log.gz 2024-09-16T13:50:31.006 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100681.log 2024-09-16T13:50:31.008 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136767.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44807.log.gz 2024-09-16T13:50:31.008 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126497.log 2024-09-16T13:50:31.008 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100681.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89635.log 2024-09-16T13:50:31.009 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126497.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125169.log 2024-09-16T13:50:31.009 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44915.log.gz 2024-09-16T13:50:31.010 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89635.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88921.log 2024-09-16T13:50:31.010 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125169.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.62562.log.gz 2024-09-16T13:50:31.010 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122882.log 2024-09-16T13:50:31.011 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88921.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118611.log 2024-09-16T13:50:31.012 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122882.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61048.log 2024-09-16T13:50:31.013 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.118611.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128259.log 2024-09-16T13:50:31.013 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61048.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.61048.log.gz 2024-09-16T13:50:31.013 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63503.log 2024-09-16T13:50:31.013 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.128259.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.128259.log.gz 2024-09-16T13:50:31.014 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77312.log 2024-09-16T13:50:31.014 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63503.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124058.log 2024-09-16T13:50:31.015 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77312.log: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.62845.log.gz 2024-09-16T13:50:31.015 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119995.log 2024-09-16T13:50:31.016 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124058.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62796.log.gz 2024-09-16T13:50:31.016 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77774.log 2024-09-16T13:50:31.016 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.119995.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106685.log 2024-09-16T13:50:31.017 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77774.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28487.log 2024-09-16T13:50:31.018 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106685.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115757.log 2024-09-16T13:50:31.018 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138003.log 2024-09-16T13:50:31.019 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115757.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30849.log 2024-09-16T13:50:31.019 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138003.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52834.log.gz 2024-09-16T13:50:31.019 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52858.log.gz 2024-09-16T13:50:31.019 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31354.log 2024-09-16T13:50:31.020 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30849.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49191.log 2024-09-16T13:50:31.021 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31354.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61051.log 2024-09-16T13:50:31.022 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49191.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53026.log.gz 2024-09-16T13:50:31.022 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53003.log.gz 2024-09-16T13:50:31.022 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132388.log 2024-09-16T13:50:31.022 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61051.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.61051.log.gz 2024-09-16T13:50:31.023 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87780.log 2024-09-16T13:50:31.024 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132388.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81036.log 2024-09-16T13:50:31.024 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87780.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108031.log 2024-09-16T13:50:31.025 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81036.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101674.log 2024-09-16T13:50:31.025 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.108031.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55768.log.gz 2024-09-16T13:50:31.025 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102301.log 2024-09-16T13:50:31.026 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101674.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114559.log 2024-09-16T13:50:31.026 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102301.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144622.log 2024-09-16T13:50:31.027 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114559.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112460.log 2024-09-16T13:50:31.027 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.144622.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144622.log.gz 2024-09-16T13:50:31.027 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44283.log 2024-09-16T13:50:31.028 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112460.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89717.log 2024-09-16T13:50:31.029 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44283.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48880.log 2024-09-16T13:50:31.029 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89717.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43593.log 2024-09-16T13:50:31.029 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.48880.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81769.log 2024-09-16T13:50:31.030 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43593.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127929.log 2024-09-16T13:50:31.030 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68530.log 2024-09-16T13:50:31.031 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127929.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56250.log.gz 2024-09-16T13:50:31.031 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90557.log 2024-09-16T13:50:31.032 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68530.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101169.log 2024-09-16T13:50:31.032 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90557.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37485.log 2024-09-16T13:50:31.033 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101169.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97476.log 2024-09-16T13:50:31.033 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37485.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68256.log.gz 2024-09-16T13:50:31.034 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105607.log 2024-09-16T13:50:31.034 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.97476.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25778.log 2024-09-16T13:50:31.035 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.105607.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105607.log.gz 2024-09-16T13:50:31.035 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87944.log 2024-09-16T13:50:31.035 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.25778.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25778.log.gz 2024-09-16T13:50:31.036 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38111.log 2024-09-16T13:50:31.036 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87944.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44431.log.gz 2024-09-16T13:50:31.036 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48520.log 2024-09-16T13:50:31.037 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.38111.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106317.log 2024-09-16T13:50:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.48520.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56165.log 2024-09-16T13:50:31.038 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106317.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100837.log 2024-09-16T13:50:31.039 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56165.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54150.log 2024-09-16T13:50:31.039 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100837.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29925.log 2024-09-16T13:50:31.040 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54150.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98991.log 2024-09-16T13:50:31.041 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29925.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67506.log.gz 2024-09-16T13:50:31.041 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77672.log.gz 2024-09-16T13:50:31.041 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133847.log 2024-09-16T13:50:31.042 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98991.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81769.log.gz 2024-09-16T13:50:31.042 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80612.log 2024-09-16T13:50:31.043 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133847.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54930.log 2024-09-16T13:50:31.043 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80612.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68889.log 2024-09-16T13:50:31.044 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54930.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28489.log 2024-09-16T13:50:31.044 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68889.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126918.log 2024-09-16T13:50:31.044 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28489.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44595.log 2024-09-16T13:50:31.045 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126918.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97362.log 2024-09-16T13:50:31.045 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44595.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.33792.log.gz 2024-09-16T13:50:31.046 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80929.log 2024-09-16T13:50:31.046 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.97362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36294.log 2024-09-16T13:50:31.046 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142958.log 2024-09-16T13:50:31.047 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36294.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64071.log.gz 2024-09-16T13:50:31.047 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112815.log 2024-09-16T13:50:31.048 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142958.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142958.log.gz 2024-09-16T13:50:31.048 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91482.log 2024-09-16T13:50:31.049 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112815.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37297.log 2024-09-16T13:50:31.049 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98372.log 2024-09-16T13:50:31.050 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.37297.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65103.log.gz 2024-09-16T13:50:31.050 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.85670.log 2024-09-16T13:50:31.051 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.98372.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29969.log 2024-09-16T13:50:31.051 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.85670.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45488.log 2024-09-16T13:50:31.052 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29969.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51491.log 2024-09-16T13:50:31.052 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45488.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133728.log 2024-09-16T13:50:31.053 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51491.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.117641.log.gz 2024-09-16T13:50:31.053 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27542.log 2024-09-16T13:50:31.054 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133728.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63464.log 2024-09-16T13:50:31.054 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.27542.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84889.log 2024-09-16T13:50:31.055 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.63464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57664.log 2024-09-16T13:50:31.055 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.84889.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121368.log 2024-09-16T13:50:31.056 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57664.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51393.log 2024-09-16T13:50:31.056 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.121368.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104808.log.gz 2024-09-16T13:50:31.056 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39805.log 2024-09-16T13:50:31.057 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51393.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142719.log 2024-09-16T13:50:31.058 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.39805.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75694.log 2024-09-16T13:50:31.058 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142719.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115083.log 2024-09-16T13:50:31.059 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75694.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.100681.log.gz 2024-09-16T13:50:31.059 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79672.log 2024-09-16T13:50:31.060 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115083.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27791.log 2024-09-16T13:50:31.061 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79672.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37777.log 2024-09-16T13:50:31.061 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% 89.6% -- replaced with /var/log/ceph/cluster1-client.admin.106317.log.gz 2024-09-16T13:50:31.061 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.97875.log.gz 2024-09-16T13:50:31.061 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.27791.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143448.log 2024-09-16T13:50:31.062 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37777.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142900.log 2024-09-16T13:50:31.062 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.143448.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135718.log 2024-09-16T13:50:31.062 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.143448.log.gz 2024-09-16T13:50:31.063 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142900.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142900.log.gz 2024-09-16T13:50:31.063 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.87944.log.gz 2024-09-16T13:50:31.063 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85410.log 2024-09-16T13:50:31.064 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135718.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56274.log 2024-09-16T13:50:31.065 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85410.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138279.log 89.9% 2024-09-16T13:50:31.066 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.101169.log.gz 2024-09-16T13:50:31.066 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56274.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138744.log 2024-09-16T13:50:31.066 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138279.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109540.log 2024-09-16T13:50:31.067 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138744.log: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.97061.log.gz 2024-09-16T13:50:31.068 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69521.log 2024-09-16T13:50:31.068 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90955.log.gz 2024-09-16T13:50:31.068 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109540.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134110.log 2024-09-16T13:50:31.068 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69521.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110689.log 2024-09-16T13:50:31.069 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134110.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136704.log 2024-09-16T13:50:31.069 INFO:teuthology.orchestra.run.smithi172.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.131460.log.gz 2024-09-16T13:50:31.070 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110689.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53248.log 2024-09-16T13:50:31.070 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136704.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143931.log 2024-09-16T13:50:31.072 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53248.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45760.log 2024-09-16T13:50:31.072 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.143931.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142358.log 2024-09-16T13:50:31.072 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.129288.log.gz 2024-09-16T13:50:31.072 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45760.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.admin.143931.log.gz 2024-09-16T13:50:31.072 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37739.log 2024-09-16T13:50:31.073 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.142358.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137062.log 2024-09-16T13:50:31.074 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122969.log 2024-09-16T13:50:31.074 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137062.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111997.log.gz 2024-09-16T13:50:31.074 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46180.log 2024-09-16T13:50:31.075 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122969.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131366.log 2024-09-16T13:50:31.075 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46180.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40283.log 2024-09-16T13:50:31.076 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.131366.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92452.log 2024-09-16T13:50:31.076 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.40283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75719.log 2024-09-16T13:50:31.077 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92452.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139542.log 2024-09-16T13:50:31.077 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75719.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137228.log 2024-09-16T13:50:31.078 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139542.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69615.log 2024-09-16T13:50:31.079 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113151.log 2024-09-16T13:50:31.079 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69615.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64441.log 2024-09-16T13:50:31.080 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105333.log.gz 2024-09-16T13:50:31.080 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113151.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98034.log 2024-09-16T13:50:31.080 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64441.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42334.log 2024-09-16T13:50:31.081 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98034.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42644.log 2024-09-16T13:50:31.081 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42334.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133128.log 2024-09-16T13:50:31.081 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98929.log.gz 2024-09-16T13:50:31.082 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42644.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28439.log 2024-09-16T13:50:31.082 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.133128.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98991.log.gz 2024-09-16T13:50:31.083 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61810.log 2024-09-16T13:50:31.083 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.28439.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30977.log 2024-09-16T13:50:31.084 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.61810.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125234.log 2024-09-16T13:50:31.085 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30977.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92072.log 2024-09-16T13:50:31.085 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77312.log.gz 2024-09-16T13:50:31.085 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125234.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97631.log 2024-09-16T13:50:31.086 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92072.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.142352.log 2024-09-16T13:50:31.086 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.97631.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90221.log 2024-09-16T13:50:31.087 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.142352.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.33712.log.gz 2024-09-16T13:50:31.087 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33503.log 2024-09-16T13:50:31.088 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90221.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142943.log 2024-09-16T13:50:31.089 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33503.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78825.log 2024-09-16T13:50:31.089 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142943.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142943.log.gz 2024-09-16T13:50:31.089 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63782.log 2024-09-16T13:50:31.090 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78825.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54300.log 2024-09-16T13:50:31.090 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138845.log 2024-09-16T13:50:31.091 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54300.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66527.log 2024-09-16T13:50:31.092 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138845.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41721.log 2024-09-16T13:50:31.092 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66527.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122317.log.gz 2024-09-16T13:50:31.093 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41047.log 2024-09-16T13:50:31.093 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41721.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64847.log 2024-09-16T13:50:31.094 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41047.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103143.log 2024-09-16T13:50:31.094 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139542.log.gz 2024-09-16T13:50:31.094 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64847.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57956.log 2024-09-16T13:50:31.095 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93378.log 2024-09-16T13:50:31.096 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57956.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84288.log 2024-09-16T13:50:31.096 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93378.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61118.log 2024-09-16T13:50:31.097 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.84288.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111277.log 2024-09-16T13:50:31.097 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61118.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97658.log 2024-09-16T13:50:31.097 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.142010.log.gz 2024-09-16T13:50:31.098 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111277.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68340.log 2024-09-16T13:50:31.098 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.97658.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94336.log 2024-09-16T13:50:31.099 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68340.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126497.log.gz 2024-09-16T13:50:31.099 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113970.log 2024-09-16T13:50:31.099 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.94336.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40695.log 2024-09-16T13:50:31.100 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113970.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79769.log 2024-09-16T13:50:31.100 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.40695.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58857.log 2024-09-16T13:50:31.102 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79769.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124185.log.gz 2024-09-16T13:50:31.102 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70089.log 2024-09-16T13:50:31.102 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58857.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88235.log 2024-09-16T13:50:31.102 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.70089.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57955.log 2024-09-16T13:50:31.103 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88235.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124329.log.gz 2024-09-16T13:50:31.103 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133759.log 2024-09-16T13:50:31.104 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57955.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137021.log 2024-09-16T13:50:31.104 INFO:teuthology.orchestra.run.smithi172.stderr: 19.0% -- replaced with /var/log/ceph/cluster1-client.admin.133128.log.gz 2024-09-16T13:50:31.105 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.133759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53461.log 2024-09-16T13:50:31.105 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59983.log 2024-09-16T13:50:31.106 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53461.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65733.log 2024-09-16T13:50:31.106 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59983.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117804.log 2024-09-16T13:50:31.107 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65733.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135507.log.gz 2024-09-16T13:50:31.107 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121603.log 2024-09-16T13:50:31.108 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117804.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107052.log 2024-09-16T13:50:31.109 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.121603.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78512.log 2024-09-16T13:50:31.109 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107052.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45442.log 2024-09-16T13:50:31.110 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78512.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46580.log 2024-09-16T13:50:31.110 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135718.log.gz 2024-09-16T13:50:31.110 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122857.log 2024-09-16T13:50:31.111 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46580.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30177.log 2024-09-16T13:50:31.112 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122857.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114981.log.gz 2024-09-16T13:50:31.112 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88057.log 2024-09-16T13:50:31.113 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30177.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79349.log 2024-09-16T13:50:31.113 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88057.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112460.log.gz 2024-09-16T13:50:31.113 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70494.log 2024-09-16T13:50:31.114 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79349.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32032.log 2024-09-16T13:50:31.115 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.70494.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66948.log 2024-09-16T13:50:31.115 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32032.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49794.log 2024-09-16T13:50:31.116 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66948.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84378.log 2024-09-16T13:50:31.116 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49794.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49794.log.gz 2024-09-16T13:50:31.117 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115336.log.gz 2024-09-16T13:50:31.117 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.36294.log.gz 2024-09-16T13:50:31.117 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55075.log 2024-09-16T13:50:31.117 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.84378.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32872.log.gz 2024-09-16T13:50:31.118 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137270.log 2024-09-16T13:50:31.118 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55075.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103338.log 2024-09-16T13:50:31.119 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137270.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118086.log 2024-09-16T13:50:31.119 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103338.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50159.log 2024-09-16T13:50:31.120 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.118086.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96511.log 2024-09-16T13:50:31.121 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.50159.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65399.log.gz 2024-09-16T13:50:31.121 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127845.log 2024-09-16T13:50:31.121 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.36676.log.gz 2024-09-16T13:50:31.121 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96511.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81984.log 2024-09-16T13:50:31.122 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127845.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118949.log 2024-09-16T13:50:31.123 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81984.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63145.log 2024-09-16T13:50:31.123 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90852.log 2024-09-16T13:50:31.124 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.63145.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81036.log.gz 2024-09-16T13:50:31.124 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115129.log 2024-09-16T13:50:31.125 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90852.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139290.log 2024-09-16T13:50:31.125 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115129.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121024.log 2024-09-16T13:50:31.126 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139290.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89695.log 2024-09-16T13:50:31.127 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.121024.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39881.log.gz 2024-09-16T13:50:31.127 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67918.log 2024-09-16T13:50:31.128 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89695.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92622.log 2024-09-16T13:50:31.128 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67918.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58334.log 2024-09-16T13:50:31.128 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92622.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69607.log 2024-09-16T13:50:31.129 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58334.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67920.log 2024-09-16T13:50:31.129 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.29673.log.gz 2024-09-16T13:50:31.130 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69607.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49822.log 2024-09-16T13:50:31.130 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67920.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62334.log 2024-09-16T13:50:31.131 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49822.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49822.log.gz 2024-09-16T13:50:31.131 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101447.log 2024-09-16T13:50:31.132 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.62334.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27316.log 2024-09-16T13:50:31.132 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37297.log.gz 2024-09-16T13:50:31.132 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101447.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99886.log 2024-09-16T13:50:31.133 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.27316.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139433.log 2024-09-16T13:50:31.133 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51491.log.gz 2024-09-16T13:50:31.134 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99886.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41045.log 2024-09-16T13:50:31.134 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139433.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66154.log 2024-09-16T13:50:31.135 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92115.log 2024-09-16T13:50:31.135 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41047.log.gz 2024-09-16T13:50:31.135 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66154.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125379.log 2024-09-16T13:50:31.136 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92115.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53088.log 2024-09-16T13:50:31.137 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117001.log 2024-09-16T13:50:31.138 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32949.log 2024-09-16T13:50:31.138 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117001.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.117001.log.gz 2024-09-16T13:50:31.138 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103549.log 2024-09-16T13:50:31.138 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138490.log 2024-09-16T13:50:31.139 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103549.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34941.log 2024-09-16T13:50:31.140 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138490.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120383.log 2024-09-16T13:50:31.140 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.34941.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34941.log.gz 2024-09-16T13:50:31.140 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87907.log 2024-09-16T13:50:31.141 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.120383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94118.log 2024-09-16T13:50:31.142 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.87907.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116646.log 2024-09-16T13:50:31.142 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.94118.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119002.log 2024-09-16T13:50:31.142 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116646.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.72701.log 2024-09-16T13:50:31.143 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.119002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109180.log 2024-09-16T13:50:31.143 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.72701.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133685.log 2024-09-16T13:50:31.144 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109180.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92875.log 2024-09-16T13:50:31.145 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133685.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67413.log 2024-09-16T13:50:31.145 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92875.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45760.log.gz 2024-09-16T13:50:31.145 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96269.log 2024-09-16T13:50:31.146 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67413.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44703.log 2024-09-16T13:50:31.147 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136559.log 2024-09-16T13:50:31.147 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44595.log.gz 2024-09-16T13:50:31.147 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44703.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52483.log 2024-09-16T13:50:31.147 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136559.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144568.log 2024-09-16T13:50:31.148 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52483.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94138.log 2024-09-16T13:50:31.149 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55075.log.gz 2024-09-16T13:50:31.149 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.144568.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144568.log.gz 2024-09-16T13:50:31.149 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.131129.log 2024-09-16T13:50:31.149 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55115.log.gz 2024-09-16T13:50:31.150 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.94138.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85002.log 2024-09-16T13:50:31.150 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.131129.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.131129.log.gz 2024-09-16T13:50:31.151 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138214.log 2024-09-16T13:50:31.151 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.85002.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133183.log 2024-09-16T13:50:31.152 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138214.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130917.log 2024-09-16T13:50:31.152 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133183.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46118.log 2024-09-16T13:50:31.153 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130917.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62380.log 2024-09-16T13:50:31.154 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46118.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.28732.log.gz 2024-09-16T13:50:31.154 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.28821.log.gz 2024-09-16T13:50:31.154 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52043.log 2024-09-16T13:50:31.155 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.62380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.63453.log 2024-09-16T13:50:31.155 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52043.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55029.log 2024-09-16T13:50:31.155 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.63453.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.63453.log.gz 2024-09-16T13:50:31.156 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97228.log 2024-09-16T13:50:31.156 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55029.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137606.log 2024-09-16T13:50:31.157 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.97228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29505.log 2024-09-16T13:50:31.157 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137606.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104178.log 2024-09-16T13:50:31.158 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29505.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136346.log 2024-09-16T13:50:31.159 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52569.log.gz 2024-09-16T13:50:31.159 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104178.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45339.log 2024-09-16T13:50:31.159 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136346.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30979.log 2024-09-16T13:50:31.159 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45339.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.86998.log 2024-09-16T13:50:31.160 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30979.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112521.log 2024-09-16T13:50:31.160 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.86998.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120109.log 2024-09-16T13:50:31.161 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66527.log.gz 2024-09-16T13:50:31.161 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112521.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34043.log 2024-09-16T13:50:31.162 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.120109.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40560.log 2024-09-16T13:50:31.162 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34043.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34253.log 2024-09-16T13:50:31.163 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40560.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129669.log 2024-09-16T13:50:31.163 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34253.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.84030.log 2024-09-16T13:50:31.164 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129669.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81459.log 2024-09-16T13:50:31.165 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.84030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45799.log 2024-09-16T13:50:31.165 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81459.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77564.log 2024-09-16T13:50:31.166 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45799.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44012.log 2024-09-16T13:50:31.166 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77564.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116203.log 2024-09-16T13:50:31.166 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44012.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127319.log 2024-09-16T13:50:31.166 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116203.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65733.log.gz 2024-09-16T13:50:31.167 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53878.log 2024-09-16T13:50:31.168 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127319.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92918.log 2024-09-16T13:50:31.169 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53878.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65397.log 2024-09-16T13:50:31.169 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92918.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69615.log.gz 2024-09-16T13:50:31.169 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116515.log 2024-09-16T13:50:31.173 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65397.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110123.log 2024-09-16T13:50:31.173 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116515.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36074.log 2024-09-16T13:50:31.174 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110123.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52529.log 2024-09-16T13:50:31.174 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36074.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111562.log 2024-09-16T13:50:31.174 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.52529.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.72674.log.gz 2024-09-16T13:50:31.174 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117029.log 2024-09-16T13:50:31.174 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111562.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79672.log.gz 2024-09-16T13:50:31.174 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50776.log 2024-09-16T13:50:31.174 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117029.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.117029.log.gz 2024-09-16T13:50:31.175 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96697.log 2024-09-16T13:50:31.175 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.50776.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40764.log 2024-09-16T13:50:31.176 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78512.log.gz 2024-09-16T13:50:31.176 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96697.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96697.log.gz 2024-09-16T13:50:31.176 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35020.log 2024-09-16T13:50:31.176 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40764.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30351.log 2024-09-16T13:50:31.177 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80612.log.gz 2024-09-16T13:50:31.177 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.35020.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86427.log 2024-09-16T13:50:31.177 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79116.log 2024-09-16T13:50:31.178 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.86427.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49789.log 2024-09-16T13:50:31.179 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79116.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46932.log 2024-09-16T13:50:31.179 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49789.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49789.log.gz 2024-09-16T13:50:31.179 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109387.log 2024-09-16T13:50:31.179 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.46932.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46932.log.gz 2024-09-16T13:50:31.180 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88786.log.gz 2024-09-16T13:50:31.180 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38837.log 2024-09-16T13:50:31.181 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.109387.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143179.log 2024-09-16T13:50:31.182 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.38837.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95520.log 2024-09-16T13:50:31.182 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.84378.log.gz 2024-09-16T13:50:31.184 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.143179.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144325.log 2024-09-16T13:50:31.185 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.95520.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68340.log.gz 2024-09-16T13:50:31.185 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125025.log 2024-09-16T13:50:31.185 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.144325.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144325.log.gz 2024-09-16T13:50:31.186 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88700.log 2024-09-16T13:50:31.187 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125025.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54172.log.gz 2024-09-16T13:50:31.187 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57745.log 2024-09-16T13:50:31.188 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88700.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57455.log 2024-09-16T13:50:31.188 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57745.log: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.143179.log.gz 2024-09-16T13:50:31.188 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110338.log 2024-09-16T13:50:31.189 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57455.log: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.85410.log.gz 2024-09-16T13:50:31.189 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139224.log 2024-09-16T13:50:31.189 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110338.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42644.log.gz 2024-09-16T13:50:31.190 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137961.log 2024-09-16T13:50:31.190 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139224.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67264.log 2024-09-16T13:50:31.191 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137961.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.86998.log.gz 2024-09-16T13:50:31.191 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127485.log 2024-09-16T13:50:31.192 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67264.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116374.log 2024-09-16T13:50:31.192 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58334.log.gz 2024-09-16T13:50:31.192 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88700.log.gz 2024-09-16T13:50:31.193 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127485.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37524.log 2024-09-16T13:50:31.193 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116374.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136323.log 2024-09-16T13:50:31.194 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37524.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.140001.log 2024-09-16T13:50:31.194 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98149.log 2024-09-16T13:50:31.195 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.140001.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.140001.log.gz 2024-09-16T13:50:31.195 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.117640.log.gz 2024-09-16T13:50:31.195 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27728.log 2024-09-16T13:50:31.197 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98149.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51936.log 2024-09-16T13:50:31.197 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115757.log.gz 2024-09-16T13:50:31.197 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.96269.log.gz/var/log/ceph/cluster1-client.mirror.27728.log: 2024-09-16T13:50:31.197 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66739.log 2024-09-16T13:50:31.198 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.51936.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134961.log 2024-09-16T13:50:31.198 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.88057.log.gz 2024-09-16T13:50:31.199 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46540.log 2024-09-16T13:50:31.199 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134961.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101862.log.gz 2024-09-16T13:50:31.200 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112670.log 2024-09-16T13:50:31.200 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46540.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94620.log 2024-09-16T13:50:31.201 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112670.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59984.log 2024-09-16T13:50:31.201 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.94620.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93378.log.gz 2024-09-16T13:50:31.201 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110613.log 2024-09-16T13:50:31.202 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59984.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55535.log 2024-09-16T13:50:31.203 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.110613.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26020.log 2024-09-16T13:50:31.203 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55535.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94138.log.gz 2024-09-16T13:50:31.203 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116688.log 2024-09-16T13:50:31.203 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.26020.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.26020.log.gz 2024-09-16T13:50:31.204 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29926.log 2024-09-16T13:50:31.205 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116688.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110817.log 2024-09-16T13:50:31.205 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29926.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.95113.log.gz 2024-09-16T13:50:31.205 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99299.log 2024-09-16T13:50:31.206 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.110817.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122814.log 2024-09-16T13:50:31.207 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99299.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42854.log 2024-09-16T13:50:31.207 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122814.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108931.log 2024-09-16T13:50:31.207 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88235.log.gz 2024-09-16T13:50:31.207 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42854.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.99504.log 2024-09-16T13:50:31.208 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.108931.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135379.log 2024-09-16T13:50:31.208 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.99504.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90392.log 2024-09-16T13:50:31.209 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87752.log 2024-09-16T13:50:31.209 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90392.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.110689.log.gz 2024-09-16T13:50:31.210 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.142352.log.gz 2024-09-16T13:50:31.210 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102093.log 2024-09-16T13:50:31.211 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87752.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48052.log 2024-09-16T13:50:31.212 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102093.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38636.log 2024-09-16T13:50:31.212 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.48052.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139013.log.gz 2024-09-16T13:50:31.212 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.121530.log 2024-09-16T13:50:31.213 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.38636.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46687.log 2024-09-16T13:50:31.213 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.121530.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87096.log 2024-09-16T13:50:31.214 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46687.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113150.log.gz 2024-09-16T13:50:31.215 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28932.log 2024-09-16T13:50:31.215 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87096.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118635.log 2024-09-16T13:50:31.216 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28932.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127969.log 2024-09-16T13:50:31.216 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118635.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116793.log 2024-09-16T13:50:31.216 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127969.log: 18.5% -- replaced with /var/log/ceph/cluster1-client.admin.133759.log.gz 2024-09-16T13:50:31.217 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72384.log 2024-09-16T13:50:31.217 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133685.log.gz 2024-09-16T13:50:31.217 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116793.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75835.log 2024-09-16T13:50:31.217 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.72384.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89905.log 2024-09-16T13:50:31.218 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75835.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58667.log 2024-09-16T13:50:31.219 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103647.log 2024-09-16T13:50:31.219 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.58667.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.58667.log.gz 2024-09-16T13:50:31.219 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50387.log 2024-09-16T13:50:31.220 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103647.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110338.log.gz 2024-09-16T13:50:31.220 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65190.log 2024-09-16T13:50:31.221 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.50387.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123221.log 2024-09-16T13:50:31.222 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65190.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92579.log.gz 2024-09-16T13:50:31.222 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90812.log 2024-09-16T13:50:31.222 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109940.log 2024-09-16T13:50:31.223 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90812.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52329.log 2024-09-16T13:50:31.224 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.109940.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143749.log 2024-09-16T13:50:31.224 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52329.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143521.log 2024-09-16T13:50:31.225 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.143749.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.128424.log.gz 2024-09-16T13:50:31.226 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57007.log 2024-09-16T13:50:31.227 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.143521.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101444.log.gz 2024-09-16T13:50:31.227 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60770.log 2024-09-16T13:50:31.228 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57007.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111303.log 2024-09-16T13:50:31.228 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.60770.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124266.log 2024-09-16T13:50:31.229 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111303.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41593.log 2024-09-16T13:50:31.230 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124266.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64893.log 2024-09-16T13:50:31.230 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41593.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130895.log 2024-09-16T13:50:31.230 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.143749.log.gz 2024-09-16T13:50:31.231 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64893.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73072.log 2024-09-16T13:50:31.231 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130895.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39195.log 2024-09-16T13:50:31.232 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73072.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132103.log 2024-09-16T13:50:31.233 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39195.log: 17.3% -- replaced with /var/log/ceph/cluster1-client.admin.121024.log.gz 2024-09-16T13:50:31.233 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89761.log 2024-09-16T13:50:31.233 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.143521.log.gz 2024-09-16T13:50:31.233 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.132103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82221.log 2024-09-16T13:50:31.234 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89761.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65188.log 2024-09-16T13:50:31.235 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.82221.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134578.log 2024-09-16T13:50:31.235 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65188.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56756.log 2024-09-16T13:50:31.236 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134578.log: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50159.log.gz 2024-09-16T13:50:31.236 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98267.log 2024-09-16T13:50:31.236 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114663.log 2024-09-16T13:50:31.237 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98267.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49128.log 2024-09-16T13:50:31.238 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114663.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39178.log 2024-09-16T13:50:31.238 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49128.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27590.log 2024-09-16T13:50:31.239 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39178.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80931.log 2024-09-16T13:50:31.239 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34253.log.gz 2024-09-16T13:50:31.240 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.27590.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143019.log 2024-09-16T13:50:31.240 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41045.log.gz 2024-09-16T13:50:31.240 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80931.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141605.log 2024-09-16T13:50:31.242 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.143019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26731.log 2024-09-16T13:50:31.242 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.63145.log.gz 2024-09-16T13:50:31.242 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141605.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84377.log 2024-09-16T13:50:31.242 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.26731.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54990.log 2024-09-16T13:50:31.243 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.84377.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109833.log 2024-09-16T13:50:31.243 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.143019.log.gz 2024-09-16T13:50:31.243 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54990.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100999.log 2024-09-16T13:50:31.244 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109833.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102850.log 2024-09-16T13:50:31.245 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100999.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44493.log 2024-09-16T13:50:31.246 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66659.log 2024-09-16T13:50:31.246 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.142339.log.gz 2024-09-16T13:50:31.246 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44493.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37950.log 2024-09-16T13:50:31.247 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66659.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66239.log 2024-09-16T13:50:31.247 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134961.log.gz 2024-09-16T13:50:31.248 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37950.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118957.log 2024-09-16T13:50:31.248 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66239.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65524.log 2024-09-16T13:50:31.249 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118957.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30561.log 2024-09-16T13:50:31.249 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65524.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.95520.log.gz 2024-09-16T13:50:31.249 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45277.log 2024-09-16T13:50:31.250 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30561.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91692.log 2024-09-16T13:50:31.251 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45277.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56545.log 2024-09-16T13:50:31.251 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79936.log 2024-09-16T13:50:31.252 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57934.log 2024-09-16T13:50:31.253 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79936.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68741.log 2024-09-16T13:50:31.253 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57934.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54424.log 2024-09-16T13:50:31.253 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68741.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130968.log 2024-09-16T13:50:31.254 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54424.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.116290.log.gz 2024-09-16T13:50:31.254 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80252.log 2024-09-16T13:50:31.254 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130968.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62220.log 2024-09-16T13:50:31.255 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80252.log: 19.0% -- replaced with /var/log/ceph/cluster2-client.admin.130901.log.gz 2024-09-16T13:50:31.255 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122219.log 2024-09-16T13:50:31.255 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.62220.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52329.log.gz 2024-09-16T13:50:31.256 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76385.log 2024-09-16T13:50:31.257 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122219.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52483.log.gz 2024-09-16T13:50:31.257 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66576.log 2024-09-16T13:50:31.257 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76385.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36289.log 2024-09-16T13:50:31.258 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66576.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119191.log 2024-09-16T13:50:31.259 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136599.log 2024-09-16T13:50:31.259 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.119191.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136431.log 2024-09-16T13:50:31.259 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45589.log.gz 2024-09-16T13:50:31.260 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136599.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56400.log 2024-09-16T13:50:31.260 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139500.log 2024-09-16T13:50:31.261 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56400.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61054.log 2024-09-16T13:50:31.261 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139500.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142534.log 2024-09-16T13:50:31.262 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61054.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.61054.log.gz 2024-09-16T13:50:31.262 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.62977.log 2024-09-16T13:50:31.263 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142534.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.142534.log.gz 2024-09-16T13:50:31.263 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56756.log.gz 2024-09-16T13:50:31.263 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113196.log 2024-09-16T13:50:31.263 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.62977.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.62977.log.gz 2024-09-16T13:50:31.264 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130077.log 2024-09-16T13:50:31.264 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44012.log.gz 2024-09-16T13:50:31.264 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113196.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133604.log 2024-09-16T13:50:31.266 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130077.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57007.log.gz 2024-09-16T13:50:31.266 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33202.log 2024-09-16T13:50:31.267 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52721.log 2024-09-16T13:50:31.267 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33202.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143863.log 2024-09-16T13:50:31.268 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52721.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116082.log 2024-09-16T13:50:31.268 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65190.log.gz 89.9% 2024-09-16T13:50:31.268 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.65188.log.gz 2024-09-16T13:50:31.269 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.143863.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66993.log 2024-09-16T13:50:31.270 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116082.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106868.log 2024-09-16T13:50:31.270 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.27542.log.gz 2024-09-16T13:50:31.270 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4%/var/log/ceph/cluster1-client.mirror.66993.log: -- replaced with /var/log/ceph/cluster1-client.mirror.27449.log.gz 2024-09-16T13:50:31.270 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45700.log 2024-09-16T13:50:31.271 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106868.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139081.log 2024-09-16T13:50:31.272 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45700.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31402.log 2024-09-16T13:50:31.272 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.27791.log.gz 2024-09-16T13:50:31.272 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139081.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89551.log 2024-09-16T13:50:31.273 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31402.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75523.log 2024-09-16T13:50:31.274 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89551.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126032.log 2024-09-16T13:50:31.274 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75523.log: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.143863.log.gz 2024-09-16T13:50:31.274 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69813.log 2024-09-16T13:50:31.275 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126032.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138533.log 2024-09-16T13:50:31.275 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41593.log.gz 2024-09-16T13:50:31.275 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69813.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65184.log 2024-09-16T13:50:31.276 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138533.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130305.log 2024-09-16T13:50:31.277 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65184.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77941.log 2024-09-16T13:50:31.277 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130305.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107579.log 2024-09-16T13:50:31.278 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68678.log 2024-09-16T13:50:31.278 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107579.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34133.log 2024-09-16T13:50:31.279 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68678.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66659.log.gz 2024-09-16T13:50:31.279 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119359.log 2024-09-16T13:50:31.280 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34133.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66739.log.gz 2024-09-16T13:50:31.280 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49181.log 2024-09-16T13:50:31.280 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.119359.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119359.log.gz 2024-09-16T13:50:31.281 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.94283.log 2024-09-16T13:50:31.282 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49181.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40764.log.gz 2024-09-16T13:50:31.282 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86911.log 2024-09-16T13:50:31.282 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.94283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120919.log 2024-09-16T13:50:31.283 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.86911.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.86911.log.gz 2024-09-16T13:50:31.283 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66993.log.gz 2024-09-16T13:50:31.283 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128271.log 2024-09-16T13:50:31.285 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.120919.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104806.log 2024-09-16T13:50:31.285 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.128271.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.128271.log.gz 2024-09-16T13:50:31.285 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66237.log 2024-09-16T13:50:31.286 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104806.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69396.log 2024-09-16T13:50:31.287 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66237.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103339.log 2024-09-16T13:50:31.287 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67264.log.gz 2024-09-16T13:50:31.287 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69396.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49609.log 2024-09-16T13:50:31.288 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103339.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114452.log 2024-09-16T13:50:31.288 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49609.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133556.log 2024-09-16T13:50:31.289 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114452.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65524.log.gz 2024-09-16T13:50:31.289 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132705.log 2024-09-16T13:50:31.290 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.133556.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67626.log 2024-09-16T13:50:31.290 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132705.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.admin.50776.log.gz 2024-09-16T13:50:31.290 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77040.log 2024-09-16T13:50:31.291 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67626.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56335.log 2024-09-16T13:50:31.291 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77040.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100199.log 2024-09-16T13:50:31.292 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56335.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109314.log 2024-09-16T13:50:31.293 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100199.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40630.log 2024-09-16T13:50:31.293 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.109314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55558.log 2024-09-16T13:50:31.294 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40630.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94941.log 2024-09-16T13:50:31.295 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55558.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109917.log 2024-09-16T13:50:31.295 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102093.log.gz 2024-09-16T13:50:31.295 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.94941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141097.log 2024-09-16T13:50:31.295 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109917.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141332.log 2024-09-16T13:50:31.297 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141097.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80252.log.gz 2024-09-16T13:50:31.297 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34255.log 2024-09-16T13:50:31.298 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141332.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78402.log.gz 2024-09-16T13:50:31.298 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96680.log 2024-09-16T13:50:31.299 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34255.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49558.log 2024-09-16T13:50:31.300 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96680.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96680.log.gz 2024-09-16T13:50:31.300 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39138.log 2024-09-16T13:50:31.300 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49558.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66363.log 2024-09-16T13:50:31.301 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.39138.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55643.log 2024-09-16T13:50:31.301 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66363.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90392.log.gz 2024-09-16T13:50:31.301 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119363.log 2024-09-16T13:50:31.302 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55643.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127550.log 2024-09-16T13:50:31.303 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.119363.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119363.log.gz 2024-09-16T13:50:31.303 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53962.log 2024-09-16T13:50:31.303 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.98149.log.gz 2024-09-16T13:50:31.303 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127550.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37534.log 2024-09-16T13:50:31.304 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53962.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.118091.log 2024-09-16T13:50:31.305 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37534.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99886.log.gz 2024-09-16T13:50:31.305 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137229.log 2024-09-16T13:50:31.306 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.118091.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99952.log.gz 2024-09-16T13:50:31.306 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128247.log 2024-09-16T13:50:31.307 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137229.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102706.log 2024-09-16T13:50:31.307 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.128247.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.128247.log.gz 2024-09-16T13:50:31.307 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39794.log 2024-09-16T13:50:31.308 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102706.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93884.log.gz 2024-09-16T13:50:31.309 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64528.log 2024-09-16T13:50:31.309 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.39794.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39794.log.gz 2024-09-16T13:50:31.309 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.120919.log.gz 2024-09-16T13:50:31.309 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141098.log 2024-09-16T13:50:31.310 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64528.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91692.log.gz 2024-09-16T13:50:31.310 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91691.log.gz 2024-09-16T13:50:31.311 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58671.log 2024-09-16T13:50:31.311 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141098.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96317.log 2024-09-16T13:50:31.312 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58671.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107255.log 2024-09-16T13:50:31.313 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96317.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102850.log.gz 2024-09-16T13:50:31.313 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103483.log 2024-09-16T13:50:31.313 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107255.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83841.log 2024-09-16T13:50:31.314 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103483.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98457.log 2024-09-16T13:50:31.315 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.83841.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105019.log 2024-09-16T13:50:31.315 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98457.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103647.log.gz 2024-09-16T13:50:31.315 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45486.log 2024-09-16T13:50:31.316 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31610.log 2024-09-16T13:50:31.317 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45486.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105206.log 2024-09-16T13:50:31.317 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31610.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.130727.log 2024-09-16T13:50:31.318 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105206.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126078.log 2024-09-16T13:50:31.319 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.130727.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116204.log 2024-09-16T13:50:31.319 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126078.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31520.log 2024-09-16T13:50:31.319 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116204.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40137.log 2024-09-16T13:50:31.320 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31520.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126706.log 2024-09-16T13:50:31.321 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.40137.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40137.log.gz 2024-09-16T13:50:31.321 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121530.log.gz 2024-09-16T13:50:31.321 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.109408.log 2024-09-16T13:50:31.321 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126706.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122158.log.gz 2024-09-16T13:50:31.321 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89340.log 2024-09-16T13:50:31.322 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.109408.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90351.log 2024-09-16T13:50:31.323 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89340.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52155.log 2024-09-16T13:50:31.323 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90351.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27135.log 2024-09-16T13:50:31.323 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52155.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32780.log 2024-09-16T13:50:31.324 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.27135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85763.log 2024-09-16T13:50:31.325 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32780.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45487.log 2024-09-16T13:50:31.325 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.85763.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121982.log 2024-09-16T13:50:31.326 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45487.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126032.log.gz 2024-09-16T13:50:31.326 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90140.log 2024-09-16T13:50:31.326 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.121982.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121982.log.gz 2024-09-16T13:50:31.327 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76778.log 2024-09-16T13:50:31.328 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127426.log 2024-09-16T13:50:31.328 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76778.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25689.log 2024-09-16T13:50:31.329 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37850.log 2024-09-16T13:50:31.329 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.25689.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25689.log.gz 2024-09-16T13:50:31.329 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68346.log 2024-09-16T13:50:31.330 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.37850.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.48992.log 2024-09-16T13:50:31.330 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68346.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143998.log 2024-09-16T13:50:31.331 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.48992.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.141331.log 2024-09-16T13:50:31.331 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.143998.log: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.143998.log.gz 2024-09-16T13:50:31.332 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126137.log 2024-09-16T13:50:31.333 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.141331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134109.log 2024-09-16T13:50:31.333 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126137.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121178.log 2024-09-16T13:50:31.333 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134109.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122219.log.gz 2024-09-16T13:50:31.334 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122093.log 2024-09-16T13:50:31.334 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.121178.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.121178.log.gz 2024-09-16T13:50:31.334 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55770.log 2024-09-16T13:50:31.335 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122093.log: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.133604.log.gz 2024-09-16T13:50:31.335 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127969.log.gz 2024-09-16T13:50:31.335 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107527.log 2024-09-16T13:50:31.337 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55770.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38796.log 2024-09-16T13:50:31.337 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107527.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.107527.log.gz 2024-09-16T13:50:31.337 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90222.log 2024-09-16T13:50:31.338 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.38796.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87844.log 2024-09-16T13:50:31.338 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137583.log 2024-09-16T13:50:31.339 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87844.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.143407.log 2024-09-16T13:50:31.339 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124959.log 2024-09-16T13:50:31.340 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137961.log.gz 2024-09-16T13:50:31.340 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.143407.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112730.log 2024-09-16T13:50:31.340 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124959.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.36074.log.gz 2024-09-16T13:50:31.341 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33838.log 2024-09-16T13:50:31.342 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112730.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95526.log 2024-09-16T13:50:31.342 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125614.log 2024-09-16T13:50:31.343 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.95526.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86515.log 2024-09-16T13:50:31.343 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125614.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94256.log 2024-09-16T13:50:31.344 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.86515.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46430.log 2024-09-16T13:50:31.344 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.94256.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.94256.log.gz 2024-09-16T13:50:31.344 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65522.log 2024-09-16T13:50:31.345 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46430.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45590.log 2024-09-16T13:50:31.346 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65522.log: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.143407.log.gz 2024-09-16T13:50:31.346 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33159.log 2024-09-16T13:50:31.346 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45590.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.120477.log 2024-09-16T13:50:31.347 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33159.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85073.log 2024-09-16T13:50:31.348 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.120477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.88093.log 2024-09-16T13:50:31.348 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28682.log 2024-09-16T13:50:31.349 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.88093.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28890.log 2024-09-16T13:50:31.349 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28682.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105629.log 2024-09-16T13:50:31.350 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28890.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127423.log 2024-09-16T13:50:31.350 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105629.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61623.log 2024-09-16T13:50:31.351 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127423.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.143475.log 2024-09-16T13:50:31.351 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.61623.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74594.log 2024-09-16T13:50:31.353 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.143475.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.143475.log.gz 2024-09-16T13:50:31.353 INFO:teuthology.orchestra.run.smithi172.stderr:gzip 18.0%-5 --verbose -- replaced with /var/log/ceph/cluster2-client.admin.37534.log.gz-- 2024-09-16T13:50:31.353 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33081.log 2024-09-16T13:50:31.353 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.74594.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74594.log.gz 2024-09-16T13:50:31.354 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31612.log 2024-09-16T13:50:31.355 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33081.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28333.log 2024-09-16T13:50:31.356 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31612.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90325.log 2024-09-16T13:50:31.356 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.28333.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135969.log 2024-09-16T13:50:31.356 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90325.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64236.log.gz 2024-09-16T13:50:31.357 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124354.log 2024-09-16T13:50:31.357 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135969.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44176.log 2024-09-16T13:50:31.358 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124354.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.109086.log 2024-09-16T13:50:31.358 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44176.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124772.log 2024-09-16T13:50:31.359 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.109086.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67579.log 2024-09-16T13:50:31.359 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124772.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99646.log 2024-09-16T13:50:31.359 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29926.log.gz 2024-09-16T13:50:31.360 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67579.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106966.log 2024-09-16T13:50:31.360 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99646.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127507.log 2024-09-16T13:50:31.361 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106966.log: 89.9%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.31520.log.gz 2024-09-16T13:50:31.361 INFO:teuthology.orchestra.run.smithi172.stderr:-5 --verbose -- /var/log/ceph/cluster2-client.admin.118936.log 2024-09-16T13:50:31.362 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127507.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.97970.log 2024-09-16T13:50:31.362 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118936.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.118936.log.gz 2024-09-16T13:50:31.362 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.36292.log.gz 2024-09-16T13:50:31.363 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107385.log 2024-09-16T13:50:31.363 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.97970.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45969.log 2024-09-16T13:50:31.364 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107385.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73085.log 2024-09-16T13:50:31.364 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45969.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139584.log 2024-09-16T13:50:31.365 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73085.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102493.log 2024-09-16T13:50:31.366 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139584.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53878.log.gz 2024-09-16T13:50:31.366 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.110756.log 2024-09-16T13:50:31.367 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102493.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142157.log 2024-09-16T13:50:31.368 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.110756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138323.log 2024-09-16T13:50:31.368 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142157.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51171.log 2024-09-16T13:50:31.368 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111499.log 2024-09-16T13:50:31.369 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.51171.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.51171.log.gz 2024-09-16T13:50:31.369 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91019.log 2024-09-16T13:50:31.370 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111499.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50045.log 2024-09-16T13:50:31.370 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79454.log 2024-09-16T13:50:31.371 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.50045.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43546.log 2024-09-16T13:50:31.372 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79454.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31400.log 2024-09-16T13:50:31.372 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43546.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55915.log 2024-09-16T13:50:31.373 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31400.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137964.log 2024-09-16T13:50:31.374 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55915.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.133915.log 2024-09-16T13:50:31.374 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137964.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.108382.log 2024-09-16T13:50:31.375 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.133915.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.21286.log 2024-09-16T13:50:31.375 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.108382.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.33414.log.gz 2024-09-16T13:50:31.376 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73065.log 2024-09-16T13:50:31.377 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.34255.log.gz 2024-09-16T13:50:31.377 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.21286.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.21286.log.gz 2024-09-16T13:50:31.377 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52287.log 2024-09-16T13:50:31.377 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73065.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73065.log.gz 2024-09-16T13:50:31.378 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130304.log 2024-09-16T13:50:31.379 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.52287.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75206.log 2024-09-16T13:50:31.379 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130304.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125042.log 2024-09-16T13:50:31.380 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.75206.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33081.log.gz 2024-09-16T13:50:31.380 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78696.log 2024-09-16T13:50:31.380 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125042.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43483.log 2024-09-16T13:50:31.381 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78696.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127400.log 2024-09-16T13:50:31.381 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43483.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68741.log.gz 2024-09-16T13:50:31.382 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135590.log 2024-09-16T13:50:31.382 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127400.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46118.log.gz 2024-09-16T13:50:31.383 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99394.log 2024-09-16T13:50:31.384 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135590.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37850.log.gz 2024-09-16T13:50:31.384 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.126792.log 2024-09-16T13:50:31.385 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99394.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90745.log 2024-09-16T13:50:31.386 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.126792.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77670.log 2024-09-16T13:50:31.386 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90745.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.125865.log 2024-09-16T13:50:31.387 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77670.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77881.log 2024-09-16T13:50:31.387 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.125865.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115970.log 2024-09-16T13:50:31.387 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77881.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.84783.log 2024-09-16T13:50:31.388 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115970.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60525.log 2024-09-16T13:50:31.389 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.84783.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.97342.log 2024-09-16T13:50:31.389 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.60525.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127930.log 2024-09-16T13:50:31.390 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.97342.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45910.log 2024-09-16T13:50:31.391 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127930.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107827.log 2024-09-16T13:50:31.391 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45910.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113677.log 2024-09-16T13:50:31.391 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43483.log.gz 2024-09-16T13:50:31.392 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107827.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30850.log 2024-09-16T13:50:31.392 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113677.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102472.log 2024-09-16T13:50:31.392 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52721.log.gz 2024-09-16T13:50:31.393 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30850.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74915.log 2024-09-16T13:50:31.394 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102472.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.108129.log 2024-09-16T13:50:31.394 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.74915.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.68318.log 2024-09-16T13:50:31.395 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.108129.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61737.log 2024-09-16T13:50:31.396 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.68318.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115462.log 2024-09-16T13:50:31.396 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.61737.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44534.log 2024-09-16T13:50:31.396 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34133.log.gz 2024-09-16T13:50:31.396 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115462.log: gzip 89.9%-5 -- replaced with /var/log/ceph/cluster1-client.mirror.55770.log.gz--verbose 2024-09-16T13:50:31.397 INFO:teuthology.orchestra.run.smithi172.stderr:-- /var/log/ceph/cluster1-client.mirror.89845.log 2024-09-16T13:50:31.397 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44534.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49267.log 2024-09-16T13:50:31.398 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89845.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77646.log 2024-09-16T13:50:31.398 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49267.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66153.log 2024-09-16T13:50:31.400 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77646.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119881.log 2024-09-16T13:50:31.400 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.39195.log.gz 2024-09-16T13:50:31.400 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66153.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109273.log 2024-09-16T13:50:31.400 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.119881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101734.log 2024-09-16T13:50:31.401 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.109273.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31192.log 2024-09-16T13:50:31.402 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101734.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30470.log 2024-09-16T13:50:31.402 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31192.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.82332.log 2024-09-16T13:50:31.403 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30470.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64733.log 2024-09-16T13:50:31.403 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.82332.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96275.log 2024-09-16T13:50:31.404 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.64733.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.58638.log 2024-09-16T13:50:31.405 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96275.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42853.log 2024-09-16T13:50:31.405 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.58638.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.58638.log.gz 2024-09-16T13:50:31.405 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66318.log 2024-09-16T13:50:31.406 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42853.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74730.log 2024-09-16T13:50:31.406 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.66318.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.95780.log 2024-09-16T13:50:31.407 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.74730.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.74730.log.gz 2024-09-16T13:50:31.407 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142012.log 2024-09-16T13:50:31.407 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.95780.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38568.log 2024-09-16T13:50:31.408 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142012.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25776.log 2024-09-16T13:50:31.408 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.38568.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103102.log 2024-09-16T13:50:31.409 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.25776.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25776.log.gz 2024-09-16T13:50:31.409 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87056.log 2024-09-16T13:50:31.410 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103102.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108569.log 2024-09-16T13:50:31.410 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.87056.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26408.log 2024-09-16T13:50:31.411 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.108569.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52153.log 2024-09-16T13:50:31.412 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.26408.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.26408.log.gz 2024-09-16T13:50:31.412 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88881.log 2024-09-16T13:50:31.412 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52153.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93713.log 2024-09-16T13:50:31.413 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55076.log 2024-09-16T13:50:31.413 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93713.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86105.log 2024-09-16T13:50:31.414 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55076.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93293.log 2024-09-16T13:50:31.415 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.86105.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84380.log 2024-09-16T13:50:31.415 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93293.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122218.log 2024-09-16T13:50:31.416 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.84380.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.49864.log 2024-09-16T13:50:31.416 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122218.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63432.log 2024-09-16T13:50:31.417 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.49864.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117805.log 2024-09-16T13:50:31.417 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63432.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127190.log 2024-09-16T13:50:31.418 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117805.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58124.log 2024-09-16T13:50:31.418 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127190.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51904.log 2024-09-16T13:50:31.419 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58124.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26356.log 2024-09-16T13:50:31.420 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98055.log 2024-09-16T13:50:31.420 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.26356.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.26356.log.gz 2024-09-16T13:50:31.420 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45759.log 2024-09-16T13:50:31.421 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.98055.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.98055.log.gz 2024-09-16T13:50:31.421 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132346.log 2024-09-16T13:50:31.421 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56124.log 2024-09-16T13:50:31.422 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.132346.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39651.log 2024-09-16T13:50:31.423 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56124.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133388.log 2024-09-16T13:50:31.423 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.39651.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130431.log 2024-09-16T13:50:31.424 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.133388.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98323.log 2024-09-16T13:50:31.424 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130431.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31736.log 2024-09-16T13:50:31.425 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44073.log 2024-09-16T13:50:31.426 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31736.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.64033.log 2024-09-16T13:50:31.426 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104678.log 2024-09-16T13:50:31.427 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.64033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98723.log 2024-09-16T13:50:31.427 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104678.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101379.log 2024-09-16T13:50:31.428 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.98723.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73491.log 2024-09-16T13:50:31.428 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101379.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43903.log 2024-09-16T13:50:31.429 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.73491.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75739.log 2024-09-16T13:50:31.429 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43903.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136197.log 2024-09-16T13:50:31.430 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.75739.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56040.log 2024-09-16T13:50:31.431 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136197.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26252.log 2024-09-16T13:50:31.431 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56040.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110604.log 2024-09-16T13:50:31.431 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.26252.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.26252.log.gz 2024-09-16T13:50:31.432 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43173.log 2024-09-16T13:50:31.433 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.110604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132164.log 2024-09-16T13:50:31.433 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43173.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113193.log 2024-09-16T13:50:31.434 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.132164.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46790.log 2024-09-16T13:50:31.434 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98834.log 2024-09-16T13:50:31.435 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.46790.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46790.log.gz 2024-09-16T13:50:31.435 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118849.log 2024-09-16T13:50:31.436 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98834.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57116.log 2024-09-16T13:50:31.436 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.118849.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139121.log 2024-09-16T13:50:31.437 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57116.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99159.log 2024-09-16T13:50:31.437 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139121.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45699.log 2024-09-16T13:50:31.438 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99159.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55745.log 2024-09-16T13:50:31.438 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45699.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31941.log 2024-09-16T13:50:31.439 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55745.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.120868.log 2024-09-16T13:50:31.440 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104278.log 2024-09-16T13:50:31.440 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.120868.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.120868.log.gz 2024-09-16T13:50:31.440 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.111502.log 2024-09-16T13:50:31.441 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104278.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104489.log 2024-09-16T13:50:31.441 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.111502.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87398.log 2024-09-16T13:50:31.442 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104489.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60513.log 2024-09-16T13:50:31.442 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.87199.log 2024-09-16T13:50:31.442 INFO:teuthology.orchestra.run.smithi172.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.99394.log.gz 2024-09-16T13:50:31.443 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.60513.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97088.log 2024-09-16T13:50:31.443 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.87199.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.87199.log.gz 2024-09-16T13:50:31.444 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134710.log 2024-09-16T13:50:31.444 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.97088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71676.log 2024-09-16T13:50:31.445 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134710.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40386.log 2024-09-16T13:50:31.446 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71676.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39247.log 2024-09-16T13:50:31.446 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.38837.log.gz 2024-09-16T13:50:31.446 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40386.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.140086.log 2024-09-16T13:50:31.447 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.39247.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85528.log 2024-09-16T13:50:31.447 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.140086.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41129.log 2024-09-16T13:50:31.448 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85528.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85528.log.gz 2024-09-16T13:50:31.448 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28930.log 2024-09-16T13:50:31.449 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41129.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116898.log 2024-09-16T13:50:31.449 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9%/var/log/ceph/cluster1-client.mirror.28930.log: -- replaced with /var/log/ceph/cluster1-client.mirror.102472.log.gz 2024-09-16T13:50:31.449 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29185.log 2024-09-16T13:50:31.450 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116898.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41866.log 2024-09-16T13:50:31.451 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29185.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67704.log 2024-09-16T13:50:31.451 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41866.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65688.log 2024-09-16T13:50:31.452 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.67704.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.97970.log.gz 2024-09-16T13:50:31.452 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77461.log 2024-09-16T13:50:31.453 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65688.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49704.log 2024-09-16T13:50:31.453 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77461.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90117.log 2024-09-16T13:50:31.454 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.94940.log.gz 2024-09-16T13:50:31.454 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49704.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.49704.log.gz 2024-09-16T13:50:31.454 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.91442.log 2024-09-16T13:50:31.455 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90117.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.119425.log 2024-09-16T13:50:31.455 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127550.log.gz 2024-09-16T13:50:31.456 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.91442.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113089.log 2024-09-16T13:50:31.457 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.119425.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116581.log 2024-09-16T13:50:31.457 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113089.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.141097.log.gz 2024-09-16T13:50:31.457 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78044.log 2024-09-16T13:50:31.458 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116581.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46849.log 2024-09-16T13:50:31.458 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78044.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134753.log 2024-09-16T13:50:31.459 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.46849.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.46849.log.gz 2024-09-16T13:50:31.459 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54385.log 2024-09-16T13:50:31.459 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134753.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.75739.log.gz 2024-09-16T13:50:31.460 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61040.log 2024-09-16T13:50:31.461 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54385.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80193.log 2024-09-16T13:50:31.461 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61040.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.61040.log.gz 2024-09-16T13:50:31.461 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135130.log 2024-09-16T13:50:31.462 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.61764.log 2024-09-16T13:50:31.463 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135130.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54929.log.gz 2024-09-16T13:50:31.464 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81917.log 2024-09-16T13:50:31.464 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.61764.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.132217.log 2024-09-16T13:50:31.464 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81917.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.98057.log 2024-09-16T13:50:31.465 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.132217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.85009.log 2024-09-16T13:50:31.466 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.98057.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.95521.log.gz 2024-09-16T13:50:31.466 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.48115.log 2024-09-16T13:50:31.466 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.85009.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59770.log 2024-09-16T13:50:31.467 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.48115.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32784.log 2024-09-16T13:50:31.467 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.109408.log.gz 2024-09-16T13:50:31.467 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59770.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45338.log 2024-09-16T13:50:31.468 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32784.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29883.log 2024-09-16T13:50:31.469 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45338.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108569.log.gz 2024-09-16T13:50:31.469 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45801.log 2024-09-16T13:50:31.469 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29883.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132613.log 2024-09-16T13:50:31.470 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45801.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34044.log 2024-09-16T13:50:31.471 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.132613.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94260.log 2024-09-16T13:50:31.471 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.34044.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102228.log 2024-09-16T13:50:31.472 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.94260.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.94260.log.gz 2024-09-16T13:50:31.472 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.121780.log 2024-09-16T13:50:31.472 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77646.log.gz 2024-09-16T13:50:31.473 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78931.log 2024-09-16T13:50:31.473 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.121780.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98349.log 2024-09-16T13:50:31.474 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78931.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51781.log 2024-09-16T13:50:31.474 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98349.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39325.log 2024-09-16T13:50:31.475 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.51781.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100328.log 2024-09-16T13:50:31.475 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39325.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102917.log 2024-09-16T13:50:31.476 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100328.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44743.log 2024-09-16T13:50:31.476 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102917.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.131989.log 2024-09-16T13:50:31.477 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36482.log 2024-09-16T13:50:31.478 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.131989.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.131936.log 2024-09-16T13:50:31.478 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55348.log 2024-09-16T13:50:31.479 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.131936.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90683.log 2024-09-16T13:50:31.479 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55348.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96259.log 2024-09-16T13:50:31.481 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90683.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68678.log.gz 2024-09-16T13:50:31.481 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83796.log 2024-09-16T13:50:31.481 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96259.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96259.log.gz 2024-09-16T13:50:31.481 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61470.log 2024-09-16T13:50:31.482 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.83796.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41418.log 2024-09-16T13:50:31.482 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61470.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.61470.log.gz 2024-09-16T13:50:31.483 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26434.log 2024-09-16T13:50:31.483 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41418.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.28238.log.gz 2024-09-16T13:50:31.484 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104619.log 2024-09-16T13:50:31.484 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.26434.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.26434.log.gz 2024-09-16T13:50:31.484 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122338.log 2024-09-16T13:50:31.485 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104619.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.71455.log 2024-09-16T13:50:31.485 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122338.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49931.log 2024-09-16T13:50:31.486 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.71455.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61060.log 2024-09-16T13:50:31.487 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.49931.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62059.log 2024-09-16T13:50:31.487 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61060.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113089.log.gz 2024-09-16T13:50:31.488 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127632.log 2024-09-16T13:50:31.488 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.62059.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51419.log 2024-09-16T13:50:31.489 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127632.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113196.log.gz 2024-09-16T13:50:31.489 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86173.log 2024-09-16T13:50:31.490 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51419.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113361.log 2024-09-16T13:50:31.491 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.86173.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76835.log 2024-09-16T13:50:31.491 INFO:teuthology.orchestra.run.smithi172.stderr: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.122093.log.gz 2024-09-16T13:50:31.491 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113361.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49019.log 2024-09-16T13:50:31.491 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76835.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63076.log 2024-09-16T13:50:31.492 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.49019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104365.log 2024-09-16T13:50:31.493 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63076.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137583.log.gz 2024-09-16T13:50:31.493 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.97745.log 2024-09-16T13:50:31.494 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104365.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.144156.log 2024-09-16T13:50:31.495 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.97745.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106867.log 2024-09-16T13:50:31.495 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139584.log.gz 2024-09-16T13:50:31.495 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.144156.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.144156.log.gz 2024-09-16T13:50:31.495 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136953.log 2024-09-16T13:50:31.496 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106867.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44596.log 2024-09-16T13:50:31.497 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136953.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53520.log 2024-09-16T13:50:31.497 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44596.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.135379.log.gz 2024-09-16T13:50:31.497 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43545.log 2024-09-16T13:50:31.498 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53520.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29674.log 2024-09-16T13:50:31.498 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.43545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44855.log 2024-09-16T13:50:31.499 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29674.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103060.log 2024-09-16T13:50:31.499 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112670.log.gz 2024-09-16T13:50:31.499 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44855.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107173.log 2024-09-16T13:50:31.500 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103060.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101171.log 2024-09-16T13:50:31.501 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107173.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122604.log 2024-09-16T13:50:31.502 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101171.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115462.log.gz 2024-09-16T13:50:31.502 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94273.log 2024-09-16T13:50:31.503 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.122604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42015.log 2024-09-16T13:50:31.503 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.94273.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.94273.log.gz 2024-09-16T13:50:31.503 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37941.log 2024-09-16T13:50:31.503 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42015.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113677.log.gz 2024-09-16T13:50:31.504 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38890.log 2024-09-16T13:50:31.504 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89804.log 2024-09-16T13:50:31.505 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.38890.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89636.log 2024-09-16T13:50:31.505 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89804.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89381.log 2024-09-16T13:50:31.506 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89636.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127632.log.gz 2024-09-16T13:50:31.506 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.119383.log 2024-09-16T13:50:31.507 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89381.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44954.log 2024-09-16T13:50:31.507 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.141098.log.gz 2024-09-16T13:50:31.507 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.119383.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.119383.log.gz 2024-09-16T13:50:31.507 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48888.log 2024-09-16T13:50:31.509 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34940.log 2024-09-16T13:50:31.510 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.48888.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50407.log 2024-09-16T13:50:31.510 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.34940.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34940.log.gz 2024-09-16T13:50:31.510 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.27728.log.gz 2024-09-16T13:50:31.510 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135903.log 2024-09-16T13:50:31.512 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.50407.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46121.log 2024-09-16T13:50:31.512 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135903.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.28930.log.gz 2024-09-16T13:50:31.512 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51553.log 2024-09-16T13:50:31.512 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46121.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69475.log 2024-09-16T13:50:31.513 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.51553.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134878.log 2024-09-16T13:50:31.513 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69475.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55558.log.gz 2024-09-16T13:50:31.513 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.100282.log 2024-09-16T13:50:31.514 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134878.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53542.log 2024-09-16T13:50:31.515 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.100282.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89424.log 2024-09-16T13:50:31.515 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53542.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113447.log 2024-09-16T13:50:31.516 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89424.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135590.log.gz 2024-09-16T13:50:31.517 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41293.log 2024-09-16T13:50:31.517 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113447.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138701.log 2024-09-16T13:50:31.518 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.41293.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44013.log 2024-09-16T13:50:31.519 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138701.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.81137.log 2024-09-16T13:50:31.519 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.44013.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126137.log.gz 2024-09-16T13:50:31.519 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.27814.log 2024-09-16T13:50:31.520 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.81137.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123975.log 2024-09-16T13:50:31.521 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.27814.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77462.log 2024-09-16T13:50:31.521 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123975.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53438.log 2024-09-16T13:50:31.522 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77462.log: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.108382.log.gz 2024-09-16T13:50:31.522 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70131.log 2024-09-16T13:50:31.522 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53438.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85408.log 2024-09-16T13:50:31.523 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.70131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112940.log 2024-09-16T13:50:31.523 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85408.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.59551.log 2024-09-16T13:50:31.524 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112940.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83621.log 2024-09-16T13:50:31.525 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.59551.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.84569.log 2024-09-16T13:50:31.525 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.83621.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127190.log.gz 2024-09-16T13:50:31.525 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39508.log 2024-09-16T13:50:31.525 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42015.log.gz 2024-09-16T13:50:31.526 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.84569.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127716.log 2024-09-16T13:50:31.527 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.39508.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65604.log 2024-09-16T13:50:31.527 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127716.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142548.log 2024-09-16T13:50:31.528 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65604.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.90141.log 2024-09-16T13:50:31.528 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134753.log.gz 2024-09-16T13:50:31.529 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46284.log 2024-09-16T13:50:31.529 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.90141.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36481.log 2024-09-16T13:50:31.529 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.46284.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.116033.log 2024-09-16T13:50:31.531 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36481.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125169.log.gz 2024-09-16T13:50:31.531 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.86374.log 2024-09-16T13:50:31.532 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.116033.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.108288.log 2024-09-16T13:50:31.532 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.86374.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.85058.log 2024-09-16T13:50:31.533 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.108288.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112772.log 2024-09-16T13:50:31.533 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.85058.log: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134538.log.gz 2024-09-16T13:50:31.533 INFO:teuthology.orchestra.run.smithi172.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.85058.log.gz 2024-09-16T13:50:31.533 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48888.log.gz 2024-09-16T13:50:31.534 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77522.log 2024-09-16T13:50:31.535 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112772.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37059.log 2024-09-16T13:50:31.535 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77522.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.21347.log 2024-09-16T13:50:31.536 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37059.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42286.log 2024-09-16T13:50:31.536 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.21347.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.21347.log.gz 2024-09-16T13:50:31.537 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75031.log 2024-09-16T13:50:31.537 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42286.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.128292.log 2024-09-16T13:50:31.538 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.75031.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78824.log 2024-09-16T13:50:31.538 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43275.log.gz 2024-09-16T13:50:31.538 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.128292.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115863.log 2024-09-16T13:50:31.539 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78824.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52106.log 2024-09-16T13:50:31.539 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115863.log: 89.9%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.66237.log.gz-5 2024-09-16T13:50:31.539 INFO:teuthology.orchestra.run.smithi172.stderr:--verbose -- /var/log/ceph/cluster1-client.mirror.78722.log 2024-09-16T13:50:31.540 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.52106.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45167.log 2024-09-16T13:50:31.541 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78722.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.120290.log 2024-09-16T13:50:31.541 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.45167.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49019.log.gz 2024-09-16T13:50:31.542 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78150.log 2024-09-16T13:50:31.542 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.120290.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118954.log 2024-09-16T13:50:31.543 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78150.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136809.log 2024-09-16T13:50:31.543 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64528.log.gz 2024-09-16T13:50:31.543 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118954.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55350.log 2024-09-16T13:50:31.544 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136809.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.50407.log.gz 2024-09-16T13:50:31.544 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.118546.log 2024-09-16T13:50:31.545 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55350.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.83620.log 2024-09-16T13:50:31.546 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.118546.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42852.log 2024-09-16T13:50:31.546 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.59551.log.gz 2024-09-16T13:50:31.546 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.83620.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.109025.log 2024-09-16T13:50:31.546 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42852.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.59770.log.gz 2024-09-16T13:50:31.547 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35163.log 2024-09-16T13:50:31.548 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.109025.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50320.log 2024-09-16T13:50:31.548 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.35163.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.98008.log 2024-09-16T13:50:31.549 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.50320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30559.log 2024-09-16T13:50:31.549 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.98008.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129764.log 2024-09-16T13:50:31.550 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30559.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75764.log 2024-09-16T13:50:31.550 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77881.log.gz 2024-09-16T13:50:31.551 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129764.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.106686.log 2024-09-16T13:50:31.551 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.75764.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.104396.log 2024-09-16T13:50:31.552 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.106686.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78781.log 2024-09-16T13:50:31.552 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.104396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.124605.log 2024-09-16T13:50:31.553 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.78781.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99986.log 2024-09-16T13:50:31.553 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.124605.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.71900.log 2024-09-16T13:50:31.554 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99986.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.37050.log 2024-09-16T13:50:31.554 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.71900.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.118959.log 2024-09-16T13:50:31.555 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.37050.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.61737.log.gz 2024-09-16T13:50:31.556 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135231.log 2024-09-16T13:50:31.556 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.118959.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.92580.log 2024-09-16T13:50:31.557 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135231.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.138424.log 2024-09-16T13:50:31.558 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.92580.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67622.log.gz 2024-09-16T13:50:31.558 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.142578.log 2024-09-16T13:50:31.559 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.138424.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65844.log 2024-09-16T13:50:31.559 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.142578.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67704.log.gz 2024-09-16T13:50:31.559 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.130890.log 2024-09-16T13:50:31.560 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65844.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102009.log 2024-09-16T13:50:31.560 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.130890.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.130890.log.gz 2024-09-16T13:50:31.561 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112671.log 2024-09-16T13:50:31.561 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102009.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136390.log 2024-09-16T13:50:31.562 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112671.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57665.log 2024-09-16T13:50:31.563 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136390.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.75362.log 2024-09-16T13:50:31.563 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57665.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33583.log 2024-09-16T13:50:31.564 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.75362.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.129535.log 2024-09-16T13:50:31.564 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33583.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136011.log 2024-09-16T13:50:31.564 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.129535.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.88465.log 2024-09-16T13:50:31.565 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136011.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40927.log 2024-09-16T13:50:31.566 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.88465.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.132477.log 2024-09-16T13:50:31.566 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40927.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.96974.log 2024-09-16T13:50:31.567 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.132477.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.117645.log 2024-09-16T13:50:31.568 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.96974.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123426.log 2024-09-16T13:50:31.568 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.117645.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25643.log 2024-09-16T13:50:31.568 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123426.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80800.log 2024-09-16T13:50:31.569 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.25643.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25643.log.gz 2024-09-16T13:50:31.569 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.49140.log 2024-09-16T13:50:31.570 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80800.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112627.log 2024-09-16T13:50:31.570 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.49140.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115781.log 2024-09-16T13:50:31.571 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112627.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73353.log 2024-09-16T13:50:31.572 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115781.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.123614.log 2024-09-16T13:50:31.572 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73353.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73353.log.gz 2024-09-16T13:50:31.573 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.110446.log 2024-09-16T13:50:31.573 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.123614.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.83621.log.gz 2024-09-16T13:50:31.573 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.96692.log 2024-09-16T13:50:31.574 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.110446.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.128983.log 2024-09-16T13:50:31.574 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.96692.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.96692.log.gz 2024-09-16T13:50:31.575 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112000.log 2024-09-16T13:50:31.575 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.128983.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77250.log.gz 2024-09-16T13:50:31.576 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31479.log 2024-09-16T13:50:31.576 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112000.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39690.log 2024-09-16T13:50:31.576 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31479.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29506.log 2024-09-16T13:50:31.577 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.39690.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.103438.log 2024-09-16T13:50:31.578 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29506.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77981.log 2024-09-16T13:50:31.578 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76800.log.gz 2024-09-16T13:50:31.578 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.103438.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53331.log 2024-09-16T13:50:31.579 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77981.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76327.log 2024-09-16T13:50:31.579 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.53331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.139541.log 2024-09-16T13:50:31.579 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76327.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119720.log 2024-09-16T13:50:31.580 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.83841.log.gz 2024-09-16T13:50:31.580 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.139541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.105332.log 2024-09-16T13:50:31.581 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.119720.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.99982.log 2024-09-16T13:50:31.582 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.105332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42960.log 2024-09-16T13:50:31.582 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.99982.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80716.log 2024-09-16T13:50:31.583 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42960.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.60314.log 2024-09-16T13:50:31.583 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80716.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.28333.log.gz 2024-09-16T13:50:31.583 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63277.log 2024-09-16T13:50:31.584 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.60314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.114407.log 2024-09-16T13:50:31.585 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63277.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76725.log 2024-09-16T13:50:31.585 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.114407.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63741.log 2024-09-16T13:50:31.587 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76725.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78150.log.gz 2024-09-16T13:50:31.587 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.89004.log 2024-09-16T13:50:31.587 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31144.log 2024-09-16T13:50:31.588 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.89004.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.80254.log 2024-09-16T13:50:31.588 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.31144.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.108797.log 2024-09-16T13:50:31.589 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.80254.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.132787.log 2024-09-16T13:50:31.590 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.108797.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.76170.log 2024-09-16T13:50:31.590 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.132787.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57642.log 2024-09-16T13:50:31.591 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.76170.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136491.log 2024-09-16T13:50:31.591 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.57642.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101021.log 2024-09-16T13:50:31.592 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136491.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40787.log 2024-09-16T13:50:31.592 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69687.log 2024-09-16T13:50:31.593 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.40787.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.50293.log 2024-09-16T13:50:31.594 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.69687.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.101101.log 2024-09-16T13:50:31.594 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.50293.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.115021.log 2024-09-16T13:50:31.595 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.101101.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33923.log 2024-09-16T13:50:31.595 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.115021.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65102.log 2024-09-16T13:50:31.596 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33923.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102228.log.gz 2024-09-16T13:50:31.596 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105617.log 2024-09-16T13:50:31.598 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.65102.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32662.log 2024-09-16T13:50:31.598 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.97745.log.gz 2024-09-16T13:50:31.598 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.105617.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105617.log.gz 2024-09-16T13:50:31.598 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.74596.log 2024-09-16T13:50:31.599 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.32662.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.133137.log 2024-09-16T13:50:31.599 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.74596.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33292.log 2024-09-16T13:50:31.600 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.133137.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105616.log 2024-09-16T13:50:31.601 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.33292.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76632.log 2024-09-16T13:50:31.601 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.105616.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105616.log.gz 2024-09-16T13:50:31.601 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29674.log.gz 2024-09-16T13:50:31.601 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.105717.log 2024-09-16T13:50:31.602 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76632.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92580.log.gz 2024-09-16T13:50:31.603 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52217.log 2024-09-16T13:50:31.603 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.105717.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.105717.log.gz 2024-09-16T13:50:31.603 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.134709.log 2024-09-16T13:50:31.604 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.52217.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.122220.log 2024-09-16T13:50:31.604 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.134709.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.113889.log 2024-09-16T13:50:31.605 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0%/var/log/ceph/cluster1-client.mirror.122220.log: -- replaced with /var/log/ceph/cluster1-client.mirror.90683.log.gz 2024-09-16T13:50:31.605 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.119948.log 2024-09-16T13:50:31.606 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.113889.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.127425.log 2024-09-16T13:50:31.607 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.119948.log: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.104278.log.gz 2024-09-16T13:50:31.607 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51036.log 2024-09-16T13:50:31.607 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.127425.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.143770.log 2024-09-16T13:50:31.608 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.51036.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.94939.log 2024-09-16T13:50:31.609 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98349.log.gz 2024-09-16T13:50:31.609 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.143770.log: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.143770.log.gz 2024-09-16T13:50:31.609 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.29841.log 2024-09-16T13:50:31.610 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.94939.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89804.log.gz 2024-09-16T13:50:31.610 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.102154.log 2024-09-16T13:50:31.611 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.29841.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116033.log.gz 2024-09-16T13:50:31.611 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93545.log 2024-09-16T13:50:31.611 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.102154.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77462.log.gz 2024-09-16T13:50:31.612 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.61044.log 2024-09-16T13:50:31.612 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93545.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.118091.log.gz 2024-09-16T13:50:31.613 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.137269.log 2024-09-16T13:50:31.613 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.61044.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.61044.log.gz 2024-09-16T13:50:31.613 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89424.log.gz 2024-09-16T13:50:31.614 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.86561.log 2024-09-16T13:50:31.615 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.137269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30978.log 2024-09-16T13:50:31.615 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91442.log.gz 2024-09-16T13:50:31.616 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.mirror.86561.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.72165.log 2024-09-16T13:50:31.616 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.30978.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58791.log 2024-09-16T13:50:31.617 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.72165.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.26512.log 2024-09-16T13:50:31.618 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58791.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.136600.log 2024-09-16T13:50:31.618 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.26512.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.26512.log.gz 2024-09-16T13:50:31.618 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57809.log.gz 2024-09-16T13:50:31.618 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70110.log 2024-09-16T13:50:31.619 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.136600.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.87579.log 2024-09-16T13:50:31.619 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.70110.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56611.log 2024-09-16T13:50:31.620 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.87579.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.107867.log 2024-09-16T13:50:31.620 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.56611.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.112186.log 2024-09-16T13:50:31.621 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.107867.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54509.log 2024-09-16T13:50:31.622 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.112186.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58019.log 2024-09-16T13:50:31.622 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118546.log.gz 2024-09-16T13:50:31.622 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.54509.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.51316.log 2024-09-16T13:50:31.623 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.58019.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63737.log 2024-09-16T13:50:31.624 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.51316.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77041.log 2024-09-16T13:50:31.624 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.63737.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42222.log 2024-09-16T13:50:31.625 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.77041.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.79455.log 2024-09-16T13:50:31.625 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137482.log.gz 2024-09-16T13:50:31.626 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.42222.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55853.log 2024-09-16T13:50:31.626 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.79455.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.93672.log 2024-09-16T13:50:31.626 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.119948.log.gz 2024-09-16T13:50:31.627 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.55853.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.73357.log 2024-09-16T13:50:31.627 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.93672.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76654.log 2024-09-16T13:50:31.628 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster2-client.admin.73357.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.73357.log.gz 2024-09-16T13:50:31.628 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.52538.log 2024-09-16T13:50:31.629 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.76654.log: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135903.log.gz 2024-09-16T13:50:31.629 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36223.log 2024-09-16T13:50:31.630 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.52538.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.135318.log 2024-09-16T13:50:31.630 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.admin.36223.log: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137269.log.gz 2024-09-16T13:50:31.632 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cluster1-client.mirror.135318.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.128983.log.gz 2024-09-16T13:50:31.633 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138701.log.gz 2024-09-16T13:50:31.634 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136953.log.gz 2024-09-16T13:50:31.638 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.28973.log.gz 2024-09-16T13:50:31.647 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90117.log.gz 2024-09-16T13:50:31.648 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.27590.log.gz 2024-09-16T13:50:31.650 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.112772.log.gz 2024-09-16T13:50:31.652 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134709.log.gz 2024-09-16T13:50:31.655 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57455.log.gz 2024-09-16T13:50:31.658 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124772.log.gz 2024-09-16T13:50:31.661 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124959.log.gz 2024-09-16T13:50:31.664 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122814.log.gz 2024-09-16T13:50:31.665 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122857.log.gz 2024-09-16T13:50:31.668 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44954.log.gz 2024-09-16T13:50:31.670 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.36223.log.gz 2024-09-16T13:50:31.674 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31940.log.gz 2024-09-16T13:50:31.677 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.61060.log.gz 2024-09-16T13:50:31.678 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.76725.log.gz 2024-09-16T13:50:31.680 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56820.log.gz 2024-09-16T13:50:31.682 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32662.log.gz 2024-09-16T13:50:31.686 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55350.log.gz 2024-09-16T13:50:31.687 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45167.log.gz 2024-09-16T13:50:31.689 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45338.log.gz 2024-09-16T13:50:31.691 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33292.log.gz 2024-09-16T13:50:31.694 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.86173.log.gz 2024-09-16T13:50:31.696 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101021.log.gz 2024-09-16T13:50:31.698 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102429.log.gz 2024-09-16T13:50:31.701 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76632.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.76654.log.gz 2024-09-16T13:50:31.701 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:31.702 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.98057.log.gz 2024-09-16T13:50:31.702 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.98008.log.gz 2024-09-16T13:50:31.705 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80254.log.gz 2024-09-16T13:50:31.709 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103271.log.gz 2024-09-16T13:50:31.711 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80716.log.gz 2024-09-16T13:50:31.712 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80800.log.gz 2024-09-16T13:50:31.714 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.94939.log.gz 2024-09-16T13:50:31.714 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.107923.log.gz 2024-09-16T13:50:31.714 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster1-client.admin.110250.log.gz 2024-09-16T13:50:31.717 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.107353.log.gz 2024-09-16T13:50:31.719 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.84027.log.gz 2024-09-16T13:50:31.721 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.38495.log.gz 2024-09-16T13:50:31.726 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.94031.log.gz 2024-09-16T13:50:31.728 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93545.log.gz 2024-09-16T13:50:31.731 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77041.log.gz 2024-09-16T13:50:31.734 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112186.log.gz 2024-09-16T13:50:31.736 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137059.log.gz 2024-09-16T13:50:31.738 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136600.log.gz 2024-09-16T13:50:31.740 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.130076.log.gz 2024-09-16T13:50:31.742 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.112783.log.gz 2024-09-16T13:50:31.744 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113553.log.gz 2024-09-16T13:50:31.744 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster1-client.admin.132954.log.gz 2024-09-16T13:50:31.746 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.130179.log.gz 2024-09-16T13:50:31.749 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126498.log.gz 2024-09-16T13:50:31.751 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127340.log.gz 2024-09-16T13:50:31.755 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.111131.log.gz 2024-09-16T13:50:31.756 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115863.log.gz 2024-09-16T13:50:31.760 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.139541.log.gz 2024-09-16T13:50:31.762 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139645.log.gz 2024-09-16T13:50:31.768 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44324.log.gz 2024-09-16T13:50:31.770 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32449.log.gz 2024-09-16T13:50:31.775 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.48556.log.gz 2024-09-16T13:50:31.777 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.48305.log.gz 2024-09-16T13:50:31.780 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30889.log.gz 2024-09-16T13:50:31.781 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30559.log.gz 2024-09-16T13:50:31.788 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57451.log.gz 2024-09-16T13:50:31.790 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66870.log.gz 2024-09-16T13:50:31.793 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69605.log.gz 2024-09-16T13:50:31.795 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56061.log.gz 2024-09-16T13:50:31.797 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69266.log.gz 2024-09-16T13:50:31.800 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.83837.log.gz 2024-09-16T13:50:31.800 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.29463.log.gz 2024-09-16T13:50:31.800 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98108.log.gz 2024-09-16T13:50:31.801 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102805.log.gz 2024-09-16T13:50:31.803 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102154.log.gz 2024-09-16T13:50:31.804 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103759.log.gz 2024-09-16T13:50:31.805 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.142012.log.gz 2024-09-16T13:50:31.805 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster2-client.admin.85081.log.gz 2024-09-16T13:50:31.810 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81265.log.gz 2024-09-16T13:50:31.812 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.59548.log.gz 2024-09-16T13:50:31.814 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56272.log.gz 2024-09-16T13:50:31.814 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90014.log.gz 2024-09-16T13:50:31.819 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92367.log.gz 2024-09-16T13:50:31.821 INFO:teuthology.orchestra.run.smithi172.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.96927.log.gz 2024-09-16T13:50:31.823 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.87884.log.gz 2024-09-16T13:50:31.828 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.117644.log.gz 2024-09-16T13:50:31.830 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120062.log.gz 2024-09-16T13:50:31.834 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136282.log.gz 2024-09-16T13:50:31.835 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121410.log.gz 2024-09-16T13:50:31.839 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65604.log.gz 2024-09-16T13:50:31.840 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.141582.log.gz 2024-09-16T13:50:31.843 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137607.log.gz 2024-09-16T13:50:31.846 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123847.log.gz 2024-09-16T13:50:31.848 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123930.log.gz 2024-09-16T13:50:31.848 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124831.log.gz 2024-09-16T13:50:31.850 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124896.log.gz 2024-09-16T13:50:31.852 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.141330.log.gz 2024-09-16T13:50:31.854 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.131165.log.gz 2024-09-16T13:50:31.856 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46009.log.gz 2024-09-16T13:50:31.860 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125504.log.gz 2024-09-16T13:50:31.864 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43381.log.gz 2024-09-16T13:50:31.870 INFO:teuthology.orchestra.run.smithi172.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.38589.log.gz 2024-09-16T13:50:31.870 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.29506.log.gz 2024-09-16T13:50:31.877 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.28645.log.gz 2024-09-16T13:50:31.881 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90181.log.gz 2024-09-16T13:50:31.883 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75460.log.gz 2024-09-16T13:50:31.885 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.129668.log.gz 2024-09-16T13:50:31.889 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103015.log.gz 2024-09-16T13:50:31.894 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56821.log.gz 2024-09-16T13:50:31.896 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.59550.log.gz 2024-09-16T13:50:31.898 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.37050.log.gz 2024-09-16T13:50:31.899 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64235.log.gz 2024-09-16T13:50:31.900 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39593.log.gz 2024-09-16T13:50:31.902 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.33582.log.gz 2024-09-16T13:50:31.903 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.33625.log.gz 2024-09-16T13:50:31.909 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56483.log.gz 2024-09-16T13:50:31.916 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81918.log.gz 2024-09-16T13:50:31.918 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79958.log.gz 2024-09-16T13:50:31.919 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102176.log.gz 2024-09-16T13:50:31.920 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81433.log.gz 2024-09-16T13:50:31.923 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79673.log.gz 2024-09-16T13:50:31.925 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.99216.log.gz 2024-09-16T13:50:31.927 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77354.log.gz 2024-09-16T13:50:31.929 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105374.log.gz 2024-09-16T13:50:31.931 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.86775.log.gz 2024-09-16T13:50:31.933 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.87201.log.gz 2024-09-16T13:50:31.935 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.87427.log.gz 2024-09-16T13:50:31.936 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.87711.log.gz 2024-09-16T13:50:31.936 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87644.log.gz 2024-09-16T13:50:31.938 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90055.log.gz 2024-09-16T13:50:31.939 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62759.log.gz 2024-09-16T13:50:31.945 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91230.log.gz 2024-09-16T13:50:31.945 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.117643.log.gz 2024-09-16T13:50:31.948 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68128.log.gz 2024-09-16T13:50:31.952 INFO:teuthology.orchestra.run.smithi172.stderr: 90.9% -- replaced with /var/log/ceph/cluster1-client.admin.120223.log.gz 2024-09-16T13:50:31.954 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137793.log.gz 2024-09-16T13:50:31.959 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113469.log.gz 2024-09-16T13:50:31.961 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126794.log.gz 2024-09-16T13:50:31.964 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138534.log.gz 2024-09-16T13:50:31.966 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136219.log.gz 2024-09-16T13:50:31.969 INFO:teuthology.orchestra.run.smithi172.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.97971.log.gz 2024-09-16T13:50:31.986 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.96281.log.gz 2024-09-16T13:50:31.989 INFO:teuthology.orchestra.run.smithi172.stderr: 56.1% -- replaced with /var/log/ceph/cluster1-client.mirror.86240.log.gz 2024-09-16T13:50:31.992 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105016.log.gz 2024-09-16T13:50:31.995 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.33793.log.gz 2024-09-16T13:50:32.000 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42285.log.gz 2024-09-16T13:50:32.004 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41913.log.gz 2024-09-16T13:50:32.006 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42075.log.gz 2024-09-16T13:50:32.007 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.48051.log.gz 2024-09-16T13:50:32.010 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.48679.log.gz 2024-09-16T13:50:32.011 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55769.log.gz 2024-09-16T13:50:32.012 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32450.log.gz 2024-09-16T13:50:32.014 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64975.log.gz 2024-09-16T13:50:32.017 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138803.log.gz 2024-09-16T13:50:32.020 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.128118.log.gz 2024-09-16T13:50:32.029 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40420.log.gz 2024-09-16T13:50:32.031 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69393.log.gz 2024-09-16T13:50:32.034 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67054.log.gz 2024-09-16T13:50:32.036 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67203.log.gz 2024-09-16T13:50:32.043 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.139435.log.gz 2024-09-16T13:50:32.043 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.83619.log.gz 2024-09-16T13:50:32.049 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.107174.log.gz 2024-09-16T13:50:32.051 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79983.log.gz 2024-09-16T13:50:32.054 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.74553.log.gz 2024-09-16T13:50:32.056 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75063.log.gz 2024-09-16T13:50:32.058 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100604.log.gz 2024-09-16T13:50:32.059 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100703.log.gz 2024-09-16T13:50:32.061 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86561.log.gz 2024-09-16T13:50:32.062 INFO:teuthology.orchestra.run.smithi172.stderr: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.73833.log.gz 2024-09-16T13:50:32.065 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80356.log.gz 2024-09-16T13:50:32.066 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89003.log.gz 2024-09-16T13:50:32.069 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102090.log.gz 2024-09-16T13:50:32.071 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.95219.log.gz 2024-09-16T13:50:32.071 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.76198.log.gz 2024-09-16T13:50:32.071 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.95831.log.gz 2024-09-16T13:50:32.117 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74709.log.gz 2024-09-16T13:50:32.118 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.86354.log.gz 2024-09-16T13:50:32.120 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89972.log.gz 2024-09-16T13:50:32.122 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.admin.109227.log.gz 2024-09-16T13:50:32.125 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126011.log.gz 2024-09-16T13:50:32.127 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137693.log.gz 2024-09-16T13:50:32.129 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.130897.log.gz 2024-09-16T13:50:32.130 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118487.log.gz 2024-09-16T13:50:32.132 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.117803.log.gz 2024-09-16T13:50:32.133 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.131802.log.gz 2024-09-16T13:50:32.143 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122432.log.gz 2024-09-16T13:50:32.144 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112352.log.gz 2024-09-16T13:50:32.146 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134327.log.gz 2024-09-16T13:50:32.149 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123489.log.gz 2024-09-16T13:50:32.151 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134052.log.gz 2024-09-16T13:50:32.152 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.118970.log.gz 2024-09-16T13:50:32.156 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126371.log.gz 2024-09-16T13:50:32.163 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29715.log.gz 2024-09-16T13:50:32.165 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92704.log.gz 2024-09-16T13:50:32.166 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98504.log.gz 2024-09-16T13:50:32.170 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.32993.log.gz 2024-09-16T13:50:32.171 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32530.log.gz 2024-09-16T13:50:32.174 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31269.log.gz 2024-09-16T13:50:32.174 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31270.log.gz 2024-09-16T13:50:32.177 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51528.log.gz 2024-09-16T13:50:32.178 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51574.log.gz 2024-09-16T13:50:32.178 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55561.log.gz 2024-09-16T13:50:32.180 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57428.log.gz 2024-09-16T13:50:32.181 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57537.log.gz 2024-09-16T13:50:32.182 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.61536.log.gz 2024-09-16T13:50:32.182 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66865.log.gz 2024-09-16T13:50:32.188 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38434.log.gz 2024-09-16T13:50:32.191 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63192.log.gz 2024-09-16T13:50:32.196 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40094.log.gz 2024-09-16T13:50:32.196 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.61070.log.gz 2024-09-16T13:50:32.199 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.74403.log.gz 2024-09-16T13:50:32.202 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99542.log.gz 2024-09-16T13:50:32.205 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.98633.log.gz 2024-09-16T13:50:32.207 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77101.log.gz 2024-09-16T13:50:32.209 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78571.log.gz 2024-09-16T13:50:32.210 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102073.log.gz 2024-09-16T13:50:32.212 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104072.log.gz 2024-09-16T13:50:32.213 INFO:teuthology.orchestra.run.smithi172.stderr: 80.5% -- replaced with /var/log/ceph/cluster1-client.mirror.86828.log.gz 2024-09-16T13:50:32.215 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.87463.log.gz 2024-09-16T13:50:32.216 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.75567.log.gz 2024-09-16T13:50:32.219 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.73397.log.gz 2024-09-16T13:50:32.222 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91317.log.gz 2024-09-16T13:50:32.223 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.86488.log.gz 2024-09-16T13:50:32.225 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.106543.log.gz 2024-09-16T13:50:32.229 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43902.log.gz 2024-09-16T13:50:32.232 INFO:teuthology.orchestra.run.smithi172.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.61992.log.gz 2024-09-16T13:50:32.234 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111540.log.gz 2024-09-16T13:50:32.236 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.118961.log.gz 2024-09-16T13:50:32.236 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.142179.log.gz 2024-09-16T13:50:32.239 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124814.log.gz 2024-09-16T13:50:32.241 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139265.log.gz 2024-09-16T13:50:32.241 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121073.log.gz 2024-09-16T13:50:32.243 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.119968.log.gz 2024-09-16T13:50:32.245 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.130075.log.gz 2024-09-16T13:50:32.255 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136009.log.gz 2024-09-16T13:50:32.257 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112628.log.gz 2024-09-16T13:50:32.258 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.110794.log.gz 2024-09-16T13:50:32.261 INFO:teuthology.orchestra.run.smithi172.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.admin.130467.log.gz 2024-09-16T13:50:32.262 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134287.log.gz 2024-09-16T13:50:32.265 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124606.log.gz 2024-09-16T13:50:32.267 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112840.log.gz 2024-09-16T13:50:32.273 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100537.log.gz 2024-09-16T13:50:32.276 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% 87.4% 87.5% 89.9% -- replaced with /var/log/ceph/cluster1-client.admin.72168.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.72164.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.71505.log.gz 2024-09-16T13:50:32.276 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:32.276 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70334.log.gz 2024-09-16T13:50:32.277 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% 90.1% 2024-09-16T13:50:32.277 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.63607.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.70335.log.gz 2024-09-16T13:50:32.277 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:32.277 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.71273.log.gz 2024-09-16T13:50:32.277 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.71456.log.gz 2024-09-16T13:50:32.277 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.72563.log.gz 2024-09-16T13:50:32.277 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.71507.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.71818.log.gz 2024-09-16T13:50:32.277 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:32.277 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58692.log.gz 2024-09-16T13:50:32.277 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70065.log.gz 2024-09-16T13:50:32.277 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.58693.log.gz 2024-09-16T13:50:32.277 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.71502.log.gz 2024-09-16T13:50:32.277 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.71435.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.71817.log.gz 2024-09-16T13:50:32.277 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:32.278 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70438.log.gz 89.6% 2024-09-16T13:50:32.278 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.71978.log.gz 2024-09-16T13:50:32.278 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70336.log.gz 2024-09-16T13:50:32.278 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.71501.log.gz 2024-09-16T13:50:32.278 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.71503.log.gz 2024-09-16T13:50:32.278 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58737.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.71506.log.gz 2024-09-16T13:50:32.278 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:32.278 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.72170.log.gz 90.0% 2024-09-16T13:50:32.278 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70349.log.gz 2024-09-16T13:50:32.278 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.72648.log.gz 2024-09-16T13:50:32.278 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41804.log.gz 2024-09-16T13:50:32.279 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53645.log.gz 2024-09-16T13:50:32.281 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68126.log.gz 2024-09-16T13:50:32.284 INFO:teuthology.orchestra.run.smithi172.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.38247.log.gz 2024-09-16T13:50:32.288 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35097.log.gz 2024-09-16T13:50:32.288 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.38381.log.gz 2024-09-16T13:50:32.290 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31821.log.gz 2024-09-16T13:50:32.293 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79304.log.gz 2024-09-16T13:50:32.296 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.72562.log.gz 2024-09-16T13:50:32.298 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64001.log.gz 2024-09-16T13:50:32.302 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81308.log.gz 2024-09-16T13:50:32.304 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101591.log.gz 2024-09-16T13:50:32.306 INFO:teuthology.orchestra.run.smithi172.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.97772.log.gz 2024-09-16T13:50:32.307 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103626.log.gz 2024-09-16T13:50:32.309 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103225.log.gz 2024-09-16T13:50:32.310 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80675.log.gz 2024-09-16T13:50:32.313 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.93085.log.gz 2024-09-16T13:50:32.316 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.93251.log.gz 2024-09-16T13:50:32.318 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103901.log.gz 2024-09-16T13:50:32.320 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91819.log.gz 2024-09-16T13:50:32.322 INFO:teuthology.orchestra.run.smithi172.stderr: 66.1% -- replaced with /var/log/ceph/cluster1-client.mirror.108610.log.gz 2024-09-16T13:50:32.326 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.95217.log.gz 2024-09-16T13:50:32.328 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52920.log.gz 2024-09-16T13:50:32.334 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.110714.log.gz 2024-09-16T13:50:32.339 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44644.log.gz 2024-09-16T13:50:32.342 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.114810.log.gz 2024-09-16T13:50:32.345 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92960.log.gz 2024-09-16T13:50:32.348 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80403.log.gz 2024-09-16T13:50:32.352 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111606.log.gz 2024-09-16T13:50:32.357 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.129939.log.gz 2024-09-16T13:50:32.361 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64653.log.gz 2024-09-16T13:50:32.364 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115192.log.gz 2024-09-16T13:50:32.366 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112042.log.gz 2024-09-16T13:50:32.367 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115442.log.gz 2024-09-16T13:50:32.367 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123846.log.gz 2024-09-16T13:50:32.371 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124456.log.gz 2024-09-16T13:50:32.372 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124481.log.gz 2024-09-16T13:50:32.372 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127296.log.gz 2024-09-16T13:50:32.373 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster1-client.admin.120870.log.gz 2024-09-16T13:50:32.375 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136852.log.gz 2024-09-16T13:50:32.377 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122431.log.gz 2024-09-16T13:50:32.378 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122538.log.gz 2024-09-16T13:50:32.381 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115822.log.gz 2024-09-16T13:50:32.385 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132703.log.gz 2024-09-16T13:50:32.385 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.38951.log.gz 2024-09-16T13:50:32.389 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139839.log.gz 2024-09-16T13:50:32.394 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66572.log.gz 2024-09-16T13:50:32.394 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41972.log.gz 2024-09-16T13:50:32.397 INFO:teuthology.orchestra.run.smithi172.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.admin.38864.log.gz 2024-09-16T13:50:32.401 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50480.log.gz 2024-09-16T13:50:32.401 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68338.log.gz 2024-09-16T13:50:32.403 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.72169.log.gz 2024-09-16T13:50:32.403 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69792.log.gz 2024-09-16T13:50:32.405 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51150.log.gz 2024-09-16T13:50:32.411 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30260.log.gz 2024-09-16T13:50:32.418 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100810.log.gz 2024-09-16T13:50:32.421 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78884.log.gz 2024-09-16T13:50:32.423 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.86913.log.gz 2024-09-16T13:50:32.423 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104322.log.gz 2024-09-16T13:50:32.424 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.87581.log.gz 2024-09-16T13:50:32.430 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.86032.log.gz 2024-09-16T13:50:32.432 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81727.log.gz 2024-09-16T13:50:32.434 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92240.log.gz 2024-09-16T13:50:32.451 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.95114.log.gz 2024-09-16T13:50:32.454 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.91903.log.gz 2024-09-16T13:50:32.457 INFO:teuthology.orchestra.run.smithi172.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.105673.log.gz 2024-09-16T13:50:32.468 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137649.log.gz 2024-09-16T13:50:32.494 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114243.log.gz 2024-09-16T13:50:32.495 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122668.log.gz 2024-09-16T13:50:32.497 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128275.log.gz 2024-09-16T13:50:32.499 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137438.log.gz 2024-09-16T13:50:32.500 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138172.log.gz 2024-09-16T13:50:32.503 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134813.log.gz 2024-09-16T13:50:32.505 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134927.log.gz 2024-09-16T13:50:32.506 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127064.log.gz 2024-09-16T13:50:32.507 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.131594.log.gz 2024-09-16T13:50:32.512 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113719.log.gz 2024-09-16T13:50:32.513 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121811.log.gz 2024-09-16T13:50:32.514 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122021.log.gz 2024-09-16T13:50:32.517 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116034.log.gz 2024-09-16T13:50:32.518 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116138.log.gz 2024-09-16T13:50:32.550 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112142.log.gz 2024-09-16T13:50:32.557 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29102.log.gz 2024-09-16T13:50:32.559 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.61851.log.gz 2024-09-16T13:50:32.563 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43276.log.gz 2024-09-16T13:50:32.564 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.72166.log.gz 2024-09-16T13:50:32.565 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41511.log.gz 2024-09-16T13:50:32.567 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41595.log.gz 2024-09-16T13:50:32.569 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63583.log.gz 2024-09-16T13:50:32.571 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67412.log.gz 2024-09-16T13:50:32.573 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103125.log.gz 2024-09-16T13:50:32.574 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69983.log.gz 2024-09-16T13:50:32.577 INFO:teuthology.orchestra.run.smithi172.stderr: 19.0% -- replaced with /var/log/ceph/cluster2-client.admin.107724.log.gz 2024-09-16T13:50:32.579 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.74081.log.gz 2024-09-16T13:50:32.581 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.87357.log.gz 2024-09-16T13:50:32.582 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.79094.log.gz 2024-09-16T13:50:32.584 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55285.log.gz 2024-09-16T13:50:32.587 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster2-client.admin.96277.log.gz 2024-09-16T13:50:32.587 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93587.log.gz 2024-09-16T13:50:32.590 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93841.log.gz 2024-09-16T13:50:32.593 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89801.log.gz 2024-09-16T13:50:32.604 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.96068.log.gz 2024-09-16T13:50:32.604 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111830.log.gz 2024-09-16T13:50:32.607 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115234.log.gz 2024-09-16T13:50:32.608 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.133105.log.gz 2024-09-16T13:50:32.611 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135802.log.gz 2024-09-16T13:50:32.633 INFO:teuthology.orchestra.run.smithi172.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.132030.log.gz 2024-09-16T13:50:32.637 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137819.log.gz 2024-09-16T13:50:32.643 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126709.log.gz 2024-09-16T13:50:32.645 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135274.log.gz 2024-09-16T13:50:32.649 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.71655.log.gz 2024-09-16T13:50:32.657 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.60313.log.gz 2024-09-16T13:50:32.659 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54025.log.gz 2024-09-16T13:50:32.662 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.117228.log.gz 2024-09-16T13:50:32.666 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119471.log.gz 2024-09-16T13:50:32.669 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51222.log.gz 2024-09-16T13:50:32.671 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.33079.log.gz 2024-09-16T13:50:32.673 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.39351.log.gz 2024-09-16T13:50:32.676 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41720.log.gz 2024-09-16T13:50:32.679 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40033.log.gz 2024-09-16T13:50:32.681 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.48707.log.gz 2024-09-16T13:50:32.683 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57514.log.gz 2024-09-16T13:50:32.694 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41089.log.gz 2024-09-16T13:50:32.695 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.59549.log.gz 2024-09-16T13:50:32.700 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54593.log.gz 2024-09-16T13:50:32.706 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45381.log.gz 2024-09-16T13:50:32.707 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45441.log.gz 2024-09-16T13:50:32.709 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45588.log.gz 2024-09-16T13:50:32.710 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.62132.log.gz 2024-09-16T13:50:32.720 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.83618.log.gz 2024-09-16T13:50:32.724 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30639.log.gz 2024-09-16T13:50:32.727 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69102.log.gz 2024-09-16T13:50:32.729 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69185.log.gz 2024-09-16T13:50:32.733 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31939.log.gz 2024-09-16T13:50:32.736 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.108771.log.gz 2024-09-16T13:50:32.742 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79241.log.gz 2024-09-16T13:50:32.745 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.100283.log.gz 2024-09-16T13:50:32.746 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99928.log.gz 2024-09-16T13:50:32.748 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.95522.log.gz 2024-09-16T13:50:32.752 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92874.log.gz 2024-09-16T13:50:32.754 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88743.log.gz 2024-09-16T13:50:32.757 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111325.log.gz 2024-09-16T13:50:32.760 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80673.log.gz 2024-09-16T13:50:32.762 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90601.log.gz 2024-09-16T13:50:32.762 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.129754.log.gz 2024-09-16T13:50:32.765 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127760.log.gz 2024-09-16T13:50:32.766 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134811.log.gz 2024-09-16T13:50:32.769 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112983.log.gz 2024-09-16T13:50:32.771 INFO:teuthology.orchestra.run.smithi172.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.132191.log.gz 2024-09-16T13:50:32.773 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112607.log.gz 2024-09-16T13:50:32.776 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67710.log.gz 2024-09-16T13:50:32.778 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66004.log.gz 2024-09-16T13:50:32.780 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29800.log.gz 2024-09-16T13:50:32.794 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62893.log.gz 2024-09-16T13:50:32.797 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.130226.log.gz 2024-09-16T13:50:32.803 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.admin.38978.log.gz 2024-09-16T13:50:32.804 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41553.log.gz 2024-09-16T13:50:32.808 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43442.log.gz 2024-09-16T13:50:32.811 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115613.log.gz 2024-09-16T13:50:32.814 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135401.log.gz 2024-09-16T13:50:32.838 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68130.log.gz 2024-09-16T13:50:32.856 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80298.log.gz 2024-09-16T13:50:32.856 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.97704.log.gz 2024-09-16T13:50:32.872 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90767.log.gz 2024-09-16T13:50:32.876 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78674.log.gz 2024-09-16T13:50:32.880 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.110735.log.gz 2024-09-16T13:50:32.889 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.50065.log.gz 2024-09-16T13:50:32.891 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40876.log.gz 2024-09-16T13:50:32.894 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.141416.log.gz 2024-09-16T13:50:32.897 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136641.log.gz 2024-09-16T13:50:32.899 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126580.log.gz 2024-09-16T13:50:32.901 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122430.log.gz 2024-09-16T13:50:32.904 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117204.log.gz 2024-09-16T13:50:32.905 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.142546.log.gz 2024-09-16T13:50:32.908 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.110994.log.gz 2024-09-16T13:50:32.912 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123135.log.gz 2024-09-16T13:50:32.915 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34345.log.gz 2024-09-16T13:50:32.919 INFO:teuthology.orchestra.run.smithi172.stderr: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.26867.log.gz 2024-09-16T13:50:32.919 INFO:teuthology.orchestra.run.smithi172.stderr: 67.2% -- replaced with /var/log/ceph/cluster2-client.admin.26989.log.gz 2024-09-16T13:50:32.923 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.60253.log.gz 2024-09-16T13:50:32.924 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.27376.log.gz 2024-09-16T13:50:32.924 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53940.log.gz 2024-09-16T13:50:32.944 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.49134.log.gz 2024-09-16T13:50:32.946 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66783.log.gz 2024-09-16T13:50:32.967 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39754.log.gz 2024-09-16T13:50:32.970 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57957.log.gz 2024-09-16T13:50:32.971 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.59723.log.gz 2024-09-16T13:50:32.973 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54444.log.gz 2024-09-16T13:50:32.976 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.28595.log.gz 2024-09-16T13:50:32.979 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69187.log.gz 2024-09-16T13:50:32.980 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69311.log.gz 2024-09-16T13:50:32.981 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69103.log.gz 2024-09-16T13:50:32.982 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63818.log.gz 2024-09-16T13:50:32.982 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69943.log.gz 2024-09-16T13:50:32.988 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63960.log.gz 2024-09-16T13:50:32.997 INFO:teuthology.orchestra.run.smithi172.stderr: 66.1% -- replaced with /var/log/ceph/cluster1-client.mirror.62401.log.gz 2024-09-16T13:50:33.004 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76885.log.gz 2024-09-16T13:50:33.007 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77141.log.gz 2024-09-16T13:50:33.053 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124396.log.gz 2024-09-16T13:50:33.057 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77204.log.gz 2024-09-16T13:50:33.060 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134853.log.gz 2024-09-16T13:50:33.062 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91271.log.gz 2024-09-16T13:50:33.069 INFO:teuthology.orchestra.run.smithi172.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.97430.log.gz 2024-09-16T13:50:33.071 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102262.log.gz 2024-09-16T13:50:33.078 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42495.log.gz 2024-09-16T13:50:33.080 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66152.log.gz 2024-09-16T13:50:33.083 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114730.log.gz 2024-09-16T13:50:33.085 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138911.log.gz 2024-09-16T13:50:33.087 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125654.log.gz 2024-09-16T13:50:33.089 INFO:teuthology.orchestra.run.smithi172.stderr: 54.9% -- replaced with /var/log/ceph/cluster1-client.mirror.26782.log.gz 2024-09-16T13:50:33.090 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135483.log.gz 2024-09-16T13:50:33.093 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113511.log.gz 2024-09-16T13:50:33.096 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68553.log.gz 2024-09-16T13:50:33.096 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46805.log.gz 2024-09-16T13:50:33.099 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124057.log.gz 2024-09-16T13:50:33.103 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65398.log.gz 2024-09-16T13:50:33.105 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.71968.log.gz 2024-09-16T13:50:33.109 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77942.log.gz 2024-09-16T13:50:33.112 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39024.log.gz 2024-09-16T13:50:33.115 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42812.log.gz 2024-09-16T13:50:33.117 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46222.log.gz 2024-09-16T13:50:33.122 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30513.log.gz 2024-09-16T13:50:33.123 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54485.log.gz 2024-09-16T13:50:33.129 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56798.log.gz 2024-09-16T13:50:33.132 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.37047.log.gz 2024-09-16T13:50:33.134 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64537.log.gz 2024-09-16T13:50:33.136 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.64713.log.gz 2024-09-16T13:50:33.137 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68129.log.gz 2024-09-16T13:50:33.138 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.27863.log.gz 2024-09-16T13:50:33.141 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.60312.log.gz 2024-09-16T13:50:33.143 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74154.log.gz 2024-09-16T13:50:33.143 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.40449.log.gz 2024-09-16T13:50:33.144 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.84559.log.gz 2024-09-16T13:50:33.145 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.49137.log.gz 2024-09-16T13:50:33.152 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52878.log.gz 2024-09-16T13:50:33.153 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.107389.log.gz 2024-09-16T13:50:33.155 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.59772.log.gz 2024-09-16T13:50:33.158 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.106316.log.gz 2024-09-16T13:50:33.160 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103335.log.gz 2024-09-16T13:50:33.167 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89087.log.gz 2024-09-16T13:50:33.168 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91735.log.gz 2024-09-16T13:50:33.169 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113552.log.gz 2024-09-16T13:50:33.171 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93042.log.gz 2024-09-16T13:50:33.173 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101123.log.gz 2024-09-16T13:50:33.181 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66866.log.gz 2024-09-16T13:50:33.184 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136978.log.gz 2024-09-16T13:50:33.193 INFO:teuthology.orchestra.run.smithi172.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.admin.84957.log.gz 2024-09-16T13:50:33.197 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69372.log.gz 2024-09-16T13:50:33.198 INFO:teuthology.orchestra.run.smithi018.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-09-16T13:50:33.205 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.109367.log.gz 2024-09-16T13:50:33.206 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134919.log.gz 2024-09-16T13:50:33.207 INFO:teuthology.orchestra.run.smithi172.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.131621.log.gz 2024-09-16T13:50:33.211 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124900.log.gz 2024-09-16T13:50:33.219 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123720.log.gz 2024-09-16T13:50:33.224 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135550.log.gz 2024-09-16T13:50:33.228 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.50434.log.gz 2024-09-16T13:50:33.232 INFO:teuthology.orchestra.run.smithi172.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.38931.log.gz 2024-09-16T13:50:33.232 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.122131.log.gz 2024-09-16T13:50:33.237 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.130713.log.gz 2024-09-16T13:50:33.241 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139331.log.gz 2024-09-16T13:50:33.245 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115044.log.gz 2024-09-16T13:50:33.247 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115151.log.gz 2024-09-16T13:50:33.249 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.62979.log.gz 2024-09-16T13:50:33.249 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128444.log.gz 2024-09-16T13:50:33.257 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.40274.log.gz 2024-09-16T13:50:33.258 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.37532.log.gz 2024-09-16T13:50:33.261 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43863.log.gz 2024-09-16T13:50:33.263 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51665.log.gz 2024-09-16T13:50:33.269 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.49978.log.gz 2024-09-16T13:50:33.279 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29295.log.gz 2024-09-16T13:50:33.281 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64976.log.gz 2024-09-16T13:50:33.285 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65897.log.gz 2024-09-16T13:50:33.285 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65943.log.gz 2024-09-16T13:50:33.289 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.84029.log.gz 2024-09-16T13:50:33.301 INFO:teuthology.orchestra.run.smithi172.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.admin.106970.log.gz 2024-09-16T13:50:33.306 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108589.log.gz 2024-09-16T13:50:33.308 INFO:teuthology.orchestra.run.smithi172.stderr: 91.4% -- replaced with /var/log/ceph/cluster2-client.admin.107945.log.gz 2024-09-16T13:50:33.308 INFO:teuthology.orchestra.run.smithi172.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.96043.log.gz 2024-09-16T13:50:33.314 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86085.log.gz 2024-09-16T13:50:33.321 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.118458.log.gz 2024-09-16T13:50:33.324 INFO:teuthology.orchestra.run.smithi172.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.105680.log.gz 2024-09-16T13:50:33.327 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121447.log.gz 2024-09-16T13:50:33.329 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115715.log.gz 2024-09-16T13:50:33.334 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136912.log.gz 2024-09-16T13:50:33.337 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80192.log.gz 2024-09-16T13:50:33.340 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138068.log.gz 2024-09-16T13:50:33.341 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138113.log.gz 2024-09-16T13:50:33.345 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54802.log.gz 2024-09-16T13:50:33.345 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.130425.log.gz 2024-09-16T13:50:33.353 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.121505.log.gz 2024-09-16T13:50:33.355 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126347.log.gz 2024-09-16T13:50:33.356 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134455.log.gz 2024-09-16T13:50:33.358 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137542.log.gz 2024-09-16T13:50:33.361 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89298.log.gz 2024-09-16T13:50:33.362 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80887.log.gz 2024-09-16T13:50:33.363 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.48286.log.gz 2024-09-16T13:50:33.368 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43692.log.gz 2024-09-16T13:50:33.370 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43172.log.gz 2024-09-16T13:50:33.370 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.49951.log.gz 2024-09-16T13:50:33.380 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64109.log.gz 2024-09-16T13:50:33.381 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.72636.log.gz 2024-09-16T13:50:33.382 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74040.log.gz 2024-09-16T13:50:33.384 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.132564.log.gz 2024-09-16T13:50:33.388 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.74683.log.gz 2024-09-16T13:50:33.392 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81138.log.gz 2024-09-16T13:50:33.394 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.49557.log.gz 2024-09-16T13:50:33.397 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54382.log.gz 2024-09-16T13:50:33.399 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.74423.log.gz 2024-09-16T13:50:33.401 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103204.log.gz 2024-09-16T13:50:33.411 INFO:teuthology.orchestra.run.smithi172.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.105651.log.gz 2024-09-16T13:50:33.416 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87097.log.gz 2024-09-16T13:50:33.425 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93335.log.gz 2024-09-16T13:50:33.429 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104112.log.gz 2024-09-16T13:50:33.430 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93714.log.gz 2024-09-16T13:50:33.432 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99185.log.gz 2024-09-16T13:50:33.436 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.110287.log.gz 2024-09-16T13:50:33.436 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.117807.log.gz 2024-09-16T13:50:33.439 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53751.log.gz 2024-09-16T13:50:33.442 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.94008.log.gz 2024-09-16T13:50:33.445 INFO:teuthology.orchestra.run.smithi172.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.119606.log.gz 2024-09-16T13:50:33.447 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137332.log.gz 2024-09-16T13:50:33.449 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137481.log.gz 2024-09-16T13:50:33.450 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124119.log.gz 2024-09-16T13:50:33.452 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136430.log.gz 2024-09-16T13:50:33.455 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123220.log.gz 2024-09-16T13:50:33.467 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126076.log.gz 2024-09-16T13:50:33.469 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.117642.log.gz 2024-09-16T13:50:33.471 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116456.log.gz 2024-09-16T13:50:33.473 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.128379.log.gz 2024-09-16T13:50:33.475 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124184.log.gz 2024-09-16T13:50:33.479 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134265.log.gz 2024-09-16T13:50:33.480 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134963.log.gz 2024-09-16T13:50:33.483 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80504.log.gz 2024-09-16T13:50:33.487 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40113.log.gz 2024-09-16T13:50:33.491 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103757.log.gz 2024-09-16T13:50:33.494 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30259.log.gz 2024-09-16T13:50:33.502 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.39796.log.gz 2024-09-16T13:50:33.502 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.51617.log.gz 2024-09-16T13:50:33.505 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32572.log.gz 2024-09-16T13:50:33.511 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46390.log.gz 2024-09-16T13:50:33.513 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49661.log.gz 2024-09-16T13:50:33.517 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34553.log.gz 2024-09-16T13:50:33.517 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34554.log.gz 2024-09-16T13:50:33.524 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.86059.log.gz 2024-09-16T13:50:33.526 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51355.log.gz 2024-09-16T13:50:33.530 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56190.log.gz 2024-09-16T13:50:33.534 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58382.log.gz 2024-09-16T13:50:33.535 INFO:teuthology.orchestra.run.smithi172.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62360.log.gz 2024-09-16T13:50:33.538 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65523.log.gz 2024-09-16T13:50:33.543 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.38294.log.gz 2024-09-16T13:50:33.547 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55012.log.gz 2024-09-16T13:50:33.548 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68633.log.gz 2024-09-16T13:50:33.552 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55981.log.gz 2024-09-16T13:50:33.556 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.48306.log.gz 2024-09-16T13:50:33.556 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80566.log.gz 2024-09-16T13:50:33.563 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81856.log.gz 2024-09-16T13:50:33.597 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.86715.log.gz 2024-09-16T13:50:33.597 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.86732.log.gz 2024-09-16T13:50:33.598 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.87617.log.gz 2024-09-16T13:50:33.641 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92200.log.gz 2024-09-16T13:50:33.643 INFO:teuthology.orchestra.run.smithi172.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.97611.log.gz 2024-09-16T13:50:33.645 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100956.log.gz 2024-09-16T13:50:33.650 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100702.log.gz 2024-09-16T13:50:33.753 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43125.log.gz 2024-09-16T13:50:33.753 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% 90.1% -- replaced with /var/log/ceph/cluster1-client.admin.39065.log.gz 2024-09-16T13:50:33.753 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.43592.log.gz 2024-09-16T13:50:33.753 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135759.log.gz 2024-09-16T13:50:33.753 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.132741.log.gz 2024-09-16T13:50:33.753 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.87178.log.gz 2024-09-16T13:50:33.753 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.87977.log.gz 2024-09-16T13:50:33.753 INFO:teuthology.orchestra.run.smithi172.stderr: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.131688.log.gz 2024-09-16T13:50:33.900 INFO:teuthology.orchestra.run.smithi018.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-09-16T13:50:33.912 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.132454.log.gz 2024-09-16T13:50:33.912 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62822.log.gz 2024-09-16T13:50:33.912 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89423.log.gz 2024-09-16T13:50:33.912 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66254.log.gz 2024-09-16T13:50:33.912 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.100117.log.gz 2024-09-16T13:50:33.912 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102175.log.gz 2024-09-16T13:50:33.912 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.133603.log.gz 2024-09-16T13:50:33.913 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.51307.log.gz 2024-09-16T13:50:33.913 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.47875.log.gz 2024-09-16T13:50:33.913 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98773.log.gz 2024-09-16T13:50:33.913 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126499.log.gz 2024-09-16T13:50:33.913 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.84855.log.gz 2024-09-16T13:50:33.913 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.127610.log.gz 2024-09-16T13:50:33.913 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53121.log.gz 2024-09-16T13:50:33.914 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74917.log.gz 2024-09-16T13:50:33.914 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.108119.log.gz 2024-09-16T13:50:33.914 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.34750.log.gz 2024-09-16T13:50:33.914 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66319.log.gz 2024-09-16T13:50:33.914 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104743.log.gz 2024-09-16T13:50:33.914 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31060.log.gz 2024-09-16T13:50:33.914 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66424.log.gz 2024-09-16T13:50:33.915 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.75871.log.gz 2024-09-16T13:50:33.915 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78316.log.gz 2024-09-16T13:50:33.915 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% 89.9% -- replaced with /var/log/ceph/cluster2-client.admin.118198.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.42120.log.gz 2024-09-16T13:50:33.915 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:33.915 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75308.log.gz 90.0% 2024-09-16T13:50:33.915 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.82025.log.gz 2024-09-16T13:50:33.915 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81200.log.gz 2024-09-16T13:50:33.915 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55916.log.gz 2024-09-16T13:50:33.915 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29634.log.gz 2024-09-16T13:50:33.915 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77773.log.gz 2024-09-16T13:50:33.915 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster2-client.admin.85075.log.gz 2024-09-16T13:50:33.915 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.86962.log.gz 2024-09-16T13:50:33.915 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87488.log.gz 2024-09-16T13:50:39.073 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 96.3% -- replaced with /var/log/ceph/cluster1-client.mirror.139080.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.119927.log.gz 2024-09-16T13:50:39.073 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.073 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68764.log.gz 2024-09-16T13:50:39.073 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster1-client.admin.121180.log.gz 2024-09-16T13:50:39.073 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125926.log.gz 2024-09-16T13:50:39.073 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68257.log.gz 2024-09-16T13:50:39.073 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58831.log.gz 2024-09-16T13:50:39.073 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94312.log.gz 2024-09-16T13:50:39.073 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77880.log.gz 2024-09-16T13:50:39.074 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63398.log.gz 2024-09-16T13:50:39.074 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.121074.log.gz 2024-09-16T13:50:39.074 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121215.log.gz 2024-09-16T13:50:39.074 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.118634.log.gz 2024-09-16T13:50:39.074 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.98965.log.gz 2024-09-16T13:50:39.074 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54698.log.gz 2024-09-16T13:50:39.074 INFO:teuthology.orchestra.run.smithi172.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.73719.log.gz 2024-09-16T13:50:39.074 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94054.log.gz 2024-09-16T13:50:39.074 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68971.log.gz 2024-09-16T13:50:39.074 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68423.log.gz 2024-09-16T13:50:39.074 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.128467.log.gz 2024-09-16T13:50:39.074 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.65626.log.gz 2024-09-16T13:50:39.074 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93209.log.gz 2024-09-16T13:50:39.074 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30723.log.gz 2024-09-16T13:50:39.075 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93671.log.gz 2024-09-16T13:50:39.075 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65814.log.gz 2024-09-16T13:50:39.075 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40877.log.gz 2024-09-16T13:50:39.075 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58375.log.gz 2024-09-16T13:50:39.075 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54024.log.gz 2024-09-16T13:50:39.075 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123824.log.gz 2024-09-16T13:50:39.075 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56670.log.gz 2024-09-16T13:50:39.075 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54801.log.gz 2024-09-16T13:50:39.075 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.28174.log.gz 2024-09-16T13:50:39.075 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41088.log.gz 2024-09-16T13:50:39.075 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103986.log.gz 2024-09-16T13:50:39.075 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45382.log.gz 2024-09-16T13:50:39.075 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30135.log.gz 2024-09-16T13:50:39.075 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.129289.log.gz 2024-09-16T13:50:39.076 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102705.log.gz 2024-09-16T13:50:39.076 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116414.log.gz 2024-09-16T13:50:39.076 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55221.log.gz 2024-09-16T13:50:39.076 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29884.log.gz 2024-09-16T13:50:39.076 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80715.log.gz 2024-09-16T13:50:39.076 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45065.log.gz 2024-09-16T13:50:39.076 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103756.log.gz 2024-09-16T13:50:39.076 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46493.log.gz 2024-09-16T13:50:39.076 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30770.log.gz 2024-09-16T13:50:39.076 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46283.log.gz 2024-09-16T13:50:39.076 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.62079.log.gz 2024-09-16T13:50:39.076 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.63300.log.gz 2024-09-16T13:50:39.076 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.48029.log.gz 2024-09-16T13:50:39.077 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103648.log.gz 2024-09-16T13:50:39.077 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125318.log.gz 2024-09-16T13:50:39.077 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67578.log.gz 2024-09-16T13:50:39.077 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69265.log.gz 2024-09-16T13:50:39.077 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116348.log.gz 2024-09-16T13:50:39.190 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88635.log.gz 2024-09-16T13:50:39.190 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137272.log.gz 2024-09-16T13:50:39.190 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.87333.log.gz 2024-09-16T13:50:39.190 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.84797.log.gz 2024-09-16T13:50:39.190 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100603.log.gz 2024-09-16T13:50:39.190 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66782.log.gz 2024-09-16T13:50:39.191 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62930.log.gz 2024-09-16T13:50:39.191 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101445.log.gz 2024-09-16T13:50:39.191 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77566.log.gz 2024-09-16T13:50:39.191 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113106.log.gz 2024-09-16T13:50:39.191 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113972.log.gz 2024-09-16T13:50:39.191 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.76715.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.77414.log.gz 2024-09-16T13:50:39.191 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.191 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86199.log.gz 2024-09-16T13:50:39.267 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91523.log.gz 2024-09-16T13:50:39.268 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79875.log.gz 2024-09-16T13:50:39.268 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139374.log.gz 2024-09-16T13:50:39.268 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121550.log.gz 2024-09-16T13:50:39.268 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103122.log.gz 2024-09-16T13:50:39.268 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.118636.log.gz 2024-09-16T13:50:39.268 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137399.log.gz 2024-09-16T13:50:39.268 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103270.log.gz 2024-09-16T13:50:39.268 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125655.log.gz 2024-09-16T13:50:39.268 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.77352.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.77351.log.gz 2024-09-16T13:50:39.268 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.268 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.141590.log.gz 2024-09-16T13:50:39.268 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.72311.log.gz 2024-09-16T13:50:39.269 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90895.log.gz 2024-09-16T13:50:39.269 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.106544.log.gz 2024-09-16T13:50:39.269 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.118199.log.gz 2024-09-16T13:50:39.269 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.133002.log.gz 2024-09-16T13:50:39.269 INFO:teuthology.orchestra.run.smithi172.stderr: 19.0% -- replaced with /var/log/ceph/cluster2-client.admin.118963.log.gz 2024-09-16T13:50:39.269 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101877.log.gz 2024-09-16T13:50:39.269 INFO:teuthology.orchestra.run.smithi172.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.96886.log.gz 2024-09-16T13:50:39.269 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31899.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.61072.log.gz 2024-09-16T13:50:39.269 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.269 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78611.log.gz 2024-09-16T13:50:39.269 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32451.log.gz 2024-09-16T13:50:39.269 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster2-client.admin.49132.log.gz 2024-09-16T13:50:39.270 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31059.log.gz 2024-09-16T13:50:39.270 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54508.log.gz 2024-09-16T13:50:39.270 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.53030.log.gz 2024-09-16T13:50:39.270 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.59987.log.gz 2024-09-16T13:50:39.270 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46643.log.gz 2024-09-16T13:50:39.270 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31310.log.gz 2024-09-16T13:50:39.270 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58082.log.gz 2024-09-16T13:50:39.270 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76884.log.gz 2024-09-16T13:50:39.270 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41438.log.gz 2024-09-16T13:50:39.270 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81012.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.80402.log.gz 2024-09-16T13:50:39.270 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.270 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.95304.log.gz 2024-09-16T13:50:39.270 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55200.log.gz 2024-09-16T13:50:39.270 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.86890.log.gz 2024-09-16T13:50:39.271 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111790.log.gz 2024-09-16T13:50:39.271 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.95769.log.gz 2024-09-16T13:50:39.271 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113630.log.gz 2024-09-16T13:50:39.271 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.119626.log.gz 2024-09-16T13:50:39.271 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93927.log.gz 2024-09-16T13:50:39.271 INFO:teuthology.orchestra.run.smithi172.stderr: 80.5% -- replaced with /var/log/ceph/cluster1-client.mirror.120649.log.gz 2024-09-16T13:50:39.271 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.132528.log.gz 2024-09-16T13:50:39.271 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.132766.log.gz 2024-09-16T13:50:39.271 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126581.log.gz 2024-09-16T13:50:39.271 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.60477.log.gz 2024-09-16T13:50:39.271 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.61068.log.gz 2024-09-16T13:50:39.271 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101842.log.gz 2024-09-16T13:50:39.271 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32570.log.gz 2024-09-16T13:50:39.272 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.47896.log.gz 2024-09-16T13:50:39.272 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.81353.log.gz 2024-09-16T13:50:39.272 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.50749.log.gz 92.0% 2024-09-16T13:50:39.272 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.59986.log.gz 2024-09-16T13:50:39.272 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78092.log.gz 2024-09-16T13:50:39.272 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.83615.log.gz 2024-09-16T13:50:39.272 INFO:teuthology.orchestra.run.smithi172.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.87272.log.gz 2024-09-16T13:50:39.272 INFO:teuthology.orchestra.run.smithi172.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.admin.50890.log.gz 2024-09-16T13:50:39.272 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53123.log.gz 2024-09-16T13:50:39.272 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64447.log.gz 2024-09-16T13:50:39.272 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136010.log.gz 2024-09-16T13:50:39.272 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.141101.log.gz 2024-09-16T13:50:39.272 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135719.log.gz 2024-09-16T13:50:39.273 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125446.log.gz 2024-09-16T13:50:39.273 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125613.log.gz 2024-09-16T13:50:39.273 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.132050.log.gz 2024-09-16T13:50:39.273 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.106718.log.gz 2024-09-16T13:50:39.273 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121747.log.gz 2024-09-16T13:50:39.273 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115232.log.gz 2024-09-16T13:50:39.273 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.102386.log.gz 2024-09-16T13:50:39.273 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30469.log.gz 2024-09-16T13:50:39.273 INFO:teuthology.orchestra.run.smithi172.stderr: 57.4% -- replaced with /var/log/ceph/cluster1-client.mirror.26833.log.gz 2024-09-16T13:50:39.273 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88922.log.gz 2024-09-16T13:50:39.273 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41973.log.gz 2024-09-16T13:50:39.273 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30562.log.gz 2024-09-16T13:50:39.273 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67787.log.gz 2024-09-16T13:50:39.273 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.97517.log.gz 2024-09-16T13:50:39.274 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79033.log.gz 2024-09-16T13:50:39.274 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79032.log.gz 2024-09-16T13:50:39.274 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101965.log.gz 2024-09-16T13:50:39.274 INFO:teuthology.orchestra.run.smithi172.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-client.admin.86401.log.gz 2024-09-16T13:50:39.274 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89382.log.gz 2024-09-16T13:50:39.274 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91399.log.gz 2024-09-16T13:50:39.274 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.84028.log.gz 2024-09-16T13:50:39.274 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101380.log.gz 2024-09-16T13:50:39.274 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.96719.log.gz 2024-09-16T13:50:39.274 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92239.log.gz 2024-09-16T13:50:39.274 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136533.log.gz 2024-09-16T13:50:39.274 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131325.log.gz 2024-09-16T13:50:39.274 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122541.log.gz 2024-09-16T13:50:39.274 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.133956.log.gz 2024-09-16T13:50:39.275 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster1-client.admin.132810.log.gz 2024-09-16T13:50:39.275 INFO:teuthology.orchestra.run.smithi172.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.133216.log.gz 2024-09-16T13:50:39.275 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115926.log.gz 2024-09-16T13:50:39.275 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125400.log.gz 2024-09-16T13:50:39.275 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124352.log.gz 2024-09-16T13:50:39.275 INFO:teuthology.orchestra.run.smithi172.stderr: 19.0% -- replaced with /var/log/ceph/cluster1-client.admin.109627.log.gz 2024-09-16T13:50:39.275 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110125.log.gz 2024-09-16T13:50:39.275 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30533.log.gz 2024-09-16T13:50:39.275 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.49142.log.gz 2024-09-16T13:50:39.275 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50615.log.gz 2024-09-16T13:50:39.275 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.141790.log.gz 2024-09-16T13:50:39.275 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43693.log.gz 2024-09-16T13:50:39.275 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66238.log.gz 2024-09-16T13:50:39.275 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67706.log.gz 2024-09-16T13:50:39.275 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67709.log.gz 2024-09-16T13:50:39.276 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.63846.log.gz 2024-09-16T13:50:39.276 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.64322.log.gz 2024-09-16T13:50:39.276 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.60309.log.gz 2024-09-16T13:50:39.276 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69730.log.gz 2024-09-16T13:50:39.276 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% 89.9% -- replaced with /var/log/ceph/cluster1-client.admin.108429.log.gz 2024-09-16T13:50:39.276 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91145.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.78906.log.gz 2024-09-16T13:50:39.276 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.276 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.91400.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.95524.log.gz 2024-09-16T13:50:39.276 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.276 INFO:teuthology.orchestra.run.smithi172.stderr: 91.2% -- replaced with /var/log/ceph/cluster1-client.admin.108999.log.gz 2024-09-16T13:50:39.276 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51730.log.gz 2024-09-16T13:50:39.276 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99481.log.gz 2024-09-16T13:50:39.276 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115233.log.gz 2024-09-16T13:50:39.277 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66657.log.gz 2024-09-16T13:50:39.277 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.84376.log.gz 2024-09-16T13:50:39.277 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster1-client.admin.121358.log.gz 2024-09-16T13:50:39.277 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121359.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.139797.log.gz 2024-09-16T13:50:39.277 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.277 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127215.log.gz 2024-09-16T13:50:39.277 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127214.log.gz 2024-09-16T13:50:39.277 INFO:teuthology.orchestra.run.smithi172.stderr: 88.1% 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.119451.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.138280.log.gz 2024-09-16T13:50:39.277 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.277 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110908.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.115149.log.gz 87.3% 2024-09-16T13:50:39.277 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.277 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.121563.log.gz 2024-09-16T13:50:39.277 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.132277.log.gz 2024-09-16T13:50:39.278 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112395.log.gz 2024-09-16T13:50:39.278 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.28644.log.gz 2024-09-16T13:50:39.278 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.28331.log.gz 2024-09-16T13:50:39.278 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.28772.log.gz 2024-09-16T13:50:39.278 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.76110.log.gz 2024-09-16T13:50:39.278 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.44114.log.gz 2024-09-16T13:50:39.278 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46601.log.gz 2024-09-16T13:50:39.278 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.60524.log.gz 2024-09-16T13:50:39.278 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52918.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.60529.log.gz 2024-09-16T13:50:39.278 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.278 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80463.log.gz 2024-09-16T13:50:39.278 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.33369.log.gz 2024-09-16T13:50:39.278 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.109294.log.gz 2024-09-16T13:50:39.278 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster2-client.admin.85412.log.gz 2024-09-16T13:50:39.279 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92705.log.gz 2024-09-16T13:50:39.279 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79832.log.gz 2024-09-16T13:50:39.279 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.72543.log.gz 2024-09-16T13:50:39.279 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113403.log.gz 2024-09-16T13:50:39.279 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.118200.log.gz 2024-09-16T13:50:39.279 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135549.log.gz 2024-09-16T13:50:39.279 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127908.log.gz 2024-09-16T13:50:39.279 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.142326.log.gz 2024-09-16T13:50:39.279 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.121680.log.gz 2024-09-16T13:50:39.279 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.132858.log.gz 2024-09-16T13:50:39.279 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134495.log.gz 2024-09-16T13:50:39.279 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133848.log.gz 2024-09-16T13:50:39.279 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134082.log.gz 2024-09-16T13:50:39.280 INFO:teuthology.orchestra.run.smithi172.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.141988.log.gz 2024-09-16T13:50:39.280 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.135128.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.129704.log.gz 2024-09-16T13:50:39.280 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.280 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91358.log.gz 2024-09-16T13:50:39.280 INFO:teuthology.orchestra.run.smithi172.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.97570.log.gz 2024-09-16T13:50:39.280 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30429.log.gz 2024-09-16T13:50:39.280 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31100.log.gz 2024-09-16T13:50:39.280 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% 89.5% -- replaced with /var/log/ceph/cluster1-client.admin.96860.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.60689.log.gz 2024-09-16T13:50:39.280 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.280 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44702.log.gz 2024-09-16T13:50:39.280 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.84296.log.gz 2024-09-16T13:50:39.280 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58292.log.gz 2024-09-16T13:50:39.280 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92157.log.gz 2024-09-16T13:50:39.281 INFO:teuthology.orchestra.run.smithi172.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.38522.log.gz 2024-09-16T13:50:39.281 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% 89.9% -- replaced with /var/log/ceph/cluster2-client.admin.95760.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.41912.log.gz 2024-09-16T13:50:39.281 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.281 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42124.log.gz 2024-09-16T13:50:39.281 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.110757.log.gz 2024-09-16T13:50:39.281 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.97041.log.gz 2024-09-16T13:50:39.281 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.91987.log.gz 2024-09-16T13:50:39.281 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.73450.log.gz 2024-09-16T13:50:39.281 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.112563.log.gz 2024-09-16T13:50:39.281 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.34634.log.gz 2024-09-16T13:50:39.281 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.110562.log.gz 2024-09-16T13:50:39.281 INFO:teuthology.orchestra.run.smithi172.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.130746.log.gz 2024-09-16T13:50:39.281 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123976.log.gz 2024-09-16T13:50:39.281 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.104909.log.gz 2024-09-16T13:50:39.282 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90979.log.gz 2024-09-16T13:50:39.282 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93798.log.gz 2024-09-16T13:50:39.282 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32156.log.gz 2024-09-16T13:50:39.282 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.62708.log.gz 2024-09-16T13:50:39.282 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32359.log.gz 2024-09-16T13:50:39.282 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64983.log.gz 2024-09-16T13:50:39.282 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64503.log.gz 2024-09-16T13:50:39.282 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.60254.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.52506.log.gz 2024-09-16T13:50:39.282 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.282 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66737.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.53541.log.gz 2024-09-16T13:50:39.282 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.282 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.77671.log.gz 2024-09-16T13:50:39.282 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.106343.log.gz 2024-09-16T13:50:39.283 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109427.log.gz 2024-09-16T13:50:39.283 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79352.log.gz 2024-09-16T13:50:39.283 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109578.log.gz 2024-09-16T13:50:39.283 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109604.log.gz 2024-09-16T13:50:39.283 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116794.log.gz 2024-09-16T13:50:39.283 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112461.log.gz 2024-09-16T13:50:39.283 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113235.log.gz 2024-09-16T13:50:39.283 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55142.log.gz 2024-09-16T13:50:39.283 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster2-client.admin.26972.log.gz 2024-09-16T13:50:39.283 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.141589.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.28708.log.gz 2024-09-16T13:50:39.283 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.283 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.84887.log.gz 2024-09-16T13:50:39.283 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.100262.log.gz 2024-09-16T13:50:39.283 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.36266.log.gz 2024-09-16T13:50:39.284 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31190.log.gz 2024-09-16T13:50:39.284 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.39118.log.gz 2024-09-16T13:50:39.284 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34466.log.gz 2024-09-16T13:50:39.284 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31519.log.gz 2024-09-16T13:50:39.284 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38776.log.gz 2024-09-16T13:50:39.284 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42433.log.gz 2024-09-16T13:50:39.284 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39459.log.gz 2024-09-16T13:50:39.284 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40387.log.gz 2024-09-16T13:50:39.284 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88657.log.gz 2024-09-16T13:50:39.284 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.96128.log.gz 2024-09-16T13:50:39.284 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90810.log.gz 2024-09-16T13:50:39.284 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.142183.log.gz 2024-09-16T13:50:39.284 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123931.log.gz 2024-09-16T13:50:39.284 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125401.log.gz 2024-09-16T13:50:39.285 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51666.log.gz 2024-09-16T13:50:39.285 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62894.log.gz 2024-09-16T13:50:39.285 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57178.log.gz 2024-09-16T13:50:39.285 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104910.log.gz 2024-09-16T13:50:39.285 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100747.log.gz 2024-09-16T13:50:39.285 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90012.log.gz 2024-09-16T13:50:39.285 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64299.log.gz 2024-09-16T13:50:39.285 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138069.log.gz 2024-09-16T13:50:39.285 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124874.log.gz 2024-09-16T13:50:39.285 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122674.log.gz 2024-09-16T13:50:39.285 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.36835.log.gz 2024-09-16T13:50:39.285 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140007.log.gz 2024-09-16T13:50:39.285 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.129345.log.gz 2024-09-16T13:50:39.285 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32029.log.gz 2024-09-16T13:50:39.286 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.38152.log.gz 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.47893.log.gz 2024-09-16T13:50:39.286 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.286 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56064.log.gz 2024-09-16T13:50:39.286 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.76161.log.gz 2024-09-16T13:50:39.286 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111892.log.gz 2024-09-16T13:50:39.286 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99559.log.gz 2024-09-16T13:50:39.286 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.99654.log.gz 2024-09-16T13:50:39.286 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99951.log.gz 2024-09-16T13:50:39.286 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79142.log.gz 2024-09-16T13:50:39.286 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.76675.log.gz 2024-09-16T13:50:39.286 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76823.log.gz 2024-09-16T13:50:39.286 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% 89.9% -- replaced with /var/log/ceph/cluster2-client.admin.107172.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.91566.log.gz 2024-09-16T13:50:39.286 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.287 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100809.log.gz 2024-09-16T13:50:39.287 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.135087.log.gz 2024-09-16T13:50:39.287 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.85603.log.gz 2024-09-16T13:50:39.287 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113887.log.gz 2024-09-16T13:50:39.287 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80505.log.gz 2024-09-16T13:50:39.287 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137063.log.gz 2024-09-16T13:50:39.287 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.133641.log.gz 2024-09-16T13:50:39.287 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.112252.log.gz 2024-09-16T13:50:39.287 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.109444.log.gz 2024-09-16T13:50:39.287 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53395.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.33289.log.gz 2024-09-16T13:50:39.287 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.287 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54570.log.gz 2024-09-16T13:50:39.287 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123344.log.gz 2024-09-16T13:50:39.288 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41191.log.gz 2024-09-16T13:50:39.288 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34346.log.gz 2024-09-16T13:50:39.288 INFO:teuthology.orchestra.run.smithi172.stderr: 88.1% 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50916.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.50206.log.gz 2024-09-16T13:50:39.288 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.288 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100913.log.gz 2024-09-16T13:50:39.288 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102492.log.gz 2024-09-16T13:50:39.288 INFO:teuthology.orchestra.run.smithi172.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.85629.log.gz 2024-09-16T13:50:39.288 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102050.log.gz 2024-09-16T13:50:39.288 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.93755.log.gz 2024-09-16T13:50:39.288 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127349.log.gz 2024-09-16T13:50:39.288 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.83835.log.gz 2024-09-16T13:50:39.288 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110495.log.gz 2024-09-16T13:50:39.288 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92282.log.gz 2024-09-16T13:50:39.288 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92323.log.gz 2024-09-16T13:50:39.289 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90600.log.gz 2024-09-16T13:50:39.289 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90768.log.gz 2024-09-16T13:50:39.289 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117115.log.gz 2024-09-16T13:50:39.289 INFO:teuthology.orchestra.run.smithi172.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.131305.log.gz 2024-09-16T13:50:39.289 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134369.log.gz 2024-09-16T13:50:39.289 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31191.log.gz 2024-09-16T13:50:39.289 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31353.log.gz 2024-09-16T13:50:39.289 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31399.log.gz 2024-09-16T13:50:39.289 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41865.log.gz 2024-09-16T13:50:39.289 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127108.log.gz 2024-09-16T13:50:39.289 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41763.log.gz 2024-09-16T13:50:39.289 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50138.log.gz 2024-09-16T13:50:39.289 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52964.log.gz 2024-09-16T13:50:39.290 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53756.log.gz 2024-09-16T13:50:39.290 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44808.log.gz 2024-09-16T13:50:39.290 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66107.log.gz 2024-09-16T13:50:39.290 INFO:teuthology.orchestra.run.smithi172.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.admin.109113.log.gz 2024-09-16T13:50:39.290 INFO:teuthology.orchestra.run.smithi172.stderr: 53.8% -- replaced with /var/log/ceph/cluster2-client.mirror.27091.log.gz 2024-09-16T13:50:39.290 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78720.log.gz 2024-09-16T13:50:39.290 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104700.log.gz 2024-09-16T13:50:39.290 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88142.log.gz 2024-09-16T13:50:39.290 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.117802.log.gz 2024-09-16T13:50:39.290 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.78401.log.gz 2024-09-16T13:50:39.290 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138239.log.gz 2024-09-16T13:50:39.290 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128012.log.gz 2024-09-16T13:50:39.290 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134430.log.gz 2024-09-16T13:50:39.290 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.113299.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.132390.log.gz 2024-09-16T13:50:39.291 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.291 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111665.log.gz 2024-09-16T13:50:39.291 INFO:teuthology.orchestra.run.smithi172.stderr: 90.9% 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.131963.log.gz 2024-09-16T13:50:39.291 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.85079.log.gz 2024-09-16T13:50:39.291 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29591.log.gz 2024-09-16T13:50:39.291 INFO:teuthology.orchestra.run.smithi172.stderr: 17.6% -- replaced with /var/log/ceph/cluster2-client.mirror.0.27146.log.gz 2024-09-16T13:50:39.291 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126288.log.gz 2024-09-16T13:50:39.291 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125716.log.gz 2024-09-16T13:50:39.291 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29464.log.gz 2024-09-16T13:50:39.291 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29425.log.gz 2024-09-16T13:50:39.291 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31480.log.gz 2024-09-16T13:50:39.291 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44532.log.gz 2024-09-16T13:50:39.291 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.48965.log.gz 2024-09-16T13:50:39.291 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56481.log.gz 2024-09-16T13:50:39.292 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.50635.log.gz 2024-09-16T13:50:39.292 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53029.log.gz 2024-09-16T13:50:39.292 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66634.log.gz 2024-09-16T13:50:39.292 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.66656.log.gz 2024-09-16T13:50:39.292 INFO:teuthology.orchestra.run.smithi172.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75834.log.gz 2024-09-16T13:50:39.292 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104618.log.gz 2024-09-16T13:50:39.292 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110497.log.gz 2024-09-16T13:50:39.292 INFO:teuthology.orchestra.run.smithi172.stderr: 17.9% -- replaced with /var/log/ceph/cluster2-client.admin.130427.log.gz 2024-09-16T13:50:39.292 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58020.log.gz 2024-09-16T13:50:39.292 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121706.log.gz 2024-09-16T13:50:39.292 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.137859.log.gz 2024-09-16T13:50:39.292 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.72542.log.gz 2024-09-16T13:50:39.292 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.36073.log.gz 2024-09-16T13:50:39.292 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38817.log.gz 2024-09-16T13:50:39.293 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.40139.log.gz 2024-09-16T13:50:39.293 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139818.log.gz 2024-09-16T13:50:39.293 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33082.log.gz 2024-09-16T13:50:39.293 INFO:teuthology.orchestra.run.smithi172.stderr: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.39287.log.gz 2024-09-16T13:50:39.293 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.63630.log.gz 2024-09-16T13:50:39.293 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.39374.log.gz 2024-09-16T13:50:39.293 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75155.log.gz 96.3% 2024-09-16T13:50:39.293 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.38340.log.gz 2024-09-16T13:50:39.293 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75632.log.gz 2024-09-16T13:50:39.293 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102010.log.gz 2024-09-16T13:50:39.293 INFO:teuthology.orchestra.run.smithi172.stderr: 16.8% -- replaced with /var/log/ceph/cluster1-client.admin.64024.log.gz 2024-09-16T13:50:39.293 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% 80.3% -- replaced with /var/log/ceph/cluster1-client.admin.95213.log.gz 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.121644.log.gz 2024-09-16T13:50:39.293 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.294 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.104532.log.gz 2024-09-16T13:50:39.294 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140006.log.gz 2024-09-16T13:50:39.294 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58792.log.gz 2024-09-16T13:50:39.294 INFO:teuthology.orchestra.run.smithi172.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.97456.log.gz 2024-09-16T13:50:39.294 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.108315.log.gz 2024-09-16T13:50:39.294 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.47226.log.gz 2024-09-16T13:50:39.294 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113761.log.gz 2024-09-16T13:50:39.294 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.120685.log.gz 2024-09-16T13:50:39.294 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120783.log.gz 2024-09-16T13:50:39.294 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40658.log.gz 2024-09-16T13:50:39.294 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123641.log.gz 2024-09-16T13:50:39.294 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31401.log.gz 2024-09-16T13:50:39.294 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53815.log.gz 2024-09-16T13:50:39.294 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51819.log.gz 2024-09-16T13:50:39.295 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43063.log.gz 2024-09-16T13:50:39.295 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102219.log.gz 2024-09-16T13:50:39.295 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100493.log.gz 89.9% 2024-09-16T13:50:39.295 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.101335.log.gz 2024-09-16T13:50:39.295 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104323.log.gz 2024-09-16T13:50:39.295 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.admin.106968.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.32319.log.gz 2024-09-16T13:50:39.295 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.295 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% 90.0% -- replaced with /var/log/ceph/cluster2-client.admin.73074.log.gz 2024-09-16T13:50:39.295 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.28082.log.gz 2024-09-16T13:50:39.295 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90267.log.gz 2024-09-16T13:50:39.295 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.118312.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.69312.log.gz 2024-09-16T13:50:39.295 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.295 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111666.log.gz 2024-09-16T13:50:39.296 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.141096.log.gz 2024-09-16T13:50:39.296 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114200.log.gz 2024-09-16T13:50:39.296 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135298.log.gz 2024-09-16T13:50:39.296 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136808.log.gz 2024-09-16T13:50:39.296 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123428.log.gz 2024-09-16T13:50:39.296 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67204.log.gz 2024-09-16T13:50:39.296 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.131708.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.142177.log.gz 2024-09-16T13:50:39.296 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.296 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29421.log.gz 2024-09-16T13:50:39.296 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43023.log.gz 2024-09-16T13:50:39.296 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.75353.log.gz 2024-09-16T13:50:39.296 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.27765.log.gz 2024-09-16T13:50:39.296 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53879.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.57808.log.gz 2024-09-16T13:50:39.297 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.297 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57324.log.gz 2024-09-16T13:50:39.297 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69606.log.gz 2024-09-16T13:50:39.297 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41510.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.30348.log.gz 2024-09-16T13:50:39.297 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.297 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.98809.log.gz 2024-09-16T13:50:39.297 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.84024.log.gz 2024-09-16T13:50:39.297 INFO:teuthology.orchestra.run.smithi172.stderr: 67.1% 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.97497.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.36480.log.gz 2024-09-16T13:50:39.297 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.297 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43694.log.gz 2024-09-16T13:50:39.297 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135506.log.gz 2024-09-16T13:50:39.297 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.124268.log.gz 2024-09-16T13:50:39.297 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.129856.log.gz 2024-09-16T13:50:39.297 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70006.log.gz 2024-09-16T13:50:39.298 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34463.log.gz 2024-09-16T13:50:39.298 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50993.log.gz 2024-09-16T13:50:39.298 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57747.log.gz 2024-09-16T13:50:39.298 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98658.log.gz 2024-09-16T13:50:39.298 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101966.log.gz 2024-09-16T13:50:39.298 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.106758.log.gz 2024-09-16T13:50:39.298 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103226.log.gz 2024-09-16T13:50:39.298 INFO:teuthology.orchestra.run.smithi172.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.76029.log.gz 2024-09-16T13:50:39.298 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.95116.log.gz 2024-09-16T13:50:39.298 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91272.log.gz 2024-09-16T13:50:39.298 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112040.log.gz 2024-09-16T13:50:39.298 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.95759.log.gz 2024-09-16T13:50:39.298 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31143.log.gz 2024-09-16T13:50:39.298 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.73070.log.gz 2024-09-16T13:50:39.299 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105230.log.gz 2024-09-16T13:50:39.299 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.106576.log.gz 2024-09-16T13:50:39.299 INFO:teuthology.orchestra.run.smithi172.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.86541.log.gz 2024-09-16T13:50:39.299 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.86146.log.gz 2024-09-16T13:50:39.299 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125863.log.gz 2024-09-16T13:50:39.299 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.117980.log.gz 2024-09-16T13:50:39.299 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.128097.log.gz 2024-09-16T13:50:39.299 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.131138.log.gz 2024-09-16T13:50:39.299 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127129.log.gz 2024-09-16T13:50:39.299 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.130223.log.gz 2024-09-16T13:50:39.299 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29382.log.gz 2024-09-16T13:50:39.299 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.36069.log.gz 2024-09-16T13:50:39.299 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81643.log.gz 2024-09-16T13:50:39.299 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104386.log.gz 2024-09-16T13:50:39.300 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55431.log.gz 2024-09-16T13:50:39.300 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44221.log.gz 2024-09-16T13:50:39.300 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44742.log.gz 2024-09-16T13:50:39.300 INFO:teuthology.orchestra.run.smithi172.stderr: 57.4% -- replaced with /var/log/ceph/cluster2-client.mirror.27074.log.gz 2024-09-16T13:50:39.300 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.27654.log.gz 2024-09-16T13:50:39.300 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39288.log.gz 2024-09-16T13:50:39.300 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127717.log.gz 2024-09-16T13:50:39.300 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55410.log.gz 2024-09-16T13:50:39.300 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127778.log.gz 2024-09-16T13:50:39.300 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138383.log.gz 2024-09-16T13:50:39.300 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53310.log.gz 2024-09-16T13:50:39.301 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.71504.log.gz 2024-09-16T13:50:39.301 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79411.log.gz 2024-09-16T13:50:39.301 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57031.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.106684.log.gz 2024-09-16T13:50:39.301 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.301 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73033.log.gz 2024-09-16T13:50:39.301 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105459.log.gz 2024-09-16T13:50:39.301 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.104397.log.gz 2024-09-16T13:50:39.301 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139710.log.gz 2024-09-16T13:50:39.301 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.115823.log.gz 2024-09-16T13:50:39.301 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.115882.log.gz 90.0% 2024-09-16T13:50:39.301 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.116073.log.gz 2024-09-16T13:50:39.301 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.141415.log.gz 2024-09-16T13:50:39.301 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.27514.log.gz 2024-09-16T13:50:39.301 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.121905.log.gz 2024-09-16T13:50:39.302 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.114600.log.gz 2024-09-16T13:50:39.302 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134031.log.gz 2024-09-16T13:50:39.302 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78990.log.gz 2024-09-16T13:50:39.302 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32152.log.gz 2024-09-16T13:50:39.302 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79664.log.gz 2024-09-16T13:50:39.302 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.34464.log.gz 2024-09-16T13:50:39.302 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43803.log.gz 2024-09-16T13:50:39.302 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43862.log.gz 2024-09-16T13:50:39.302 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.28553.log.gz 2024-09-16T13:50:39.302 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45911.log.gz 2024-09-16T13:50:39.302 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48841.log.gz 2024-09-16T13:50:39.302 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.48559.log.gz 2024-09-16T13:50:39.302 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64369.log.gz 2024-09-16T13:50:39.303 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.102364.log.gz 2024-09-16T13:50:39.303 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.96813.log.gz 2024-09-16T13:50:39.303 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104699.log.gz 2024-09-16T13:50:39.303 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% 89.9% -- replaced with /var/log/ceph/cluster2-client.admin.60530.log.gz 2024-09-16T13:50:39.303 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.31942.log.gz 2024-09-16T13:50:39.303 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.96746.log.gz 2024-09-16T13:50:39.303 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92831.log.gz 2024-09-16T13:50:39.303 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90391.log.gz 2024-09-16T13:50:39.303 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123909.log.gz 2024-09-16T13:50:39.303 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125821.log.gz 2024-09-16T13:50:39.303 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.72167.log.gz 2024-09-16T13:50:39.304 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31900.log.gz 2024-09-16T13:50:39.304 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34985.log.gz 2024-09-16T13:50:39.304 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.49500.log.gz 2024-09-16T13:50:39.304 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53604.log.gz 91.8% 2024-09-16T13:50:39.304 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.48713.log.gz 2024-09-16T13:50:39.304 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81037.log.gz 2024-09-16T13:50:39.304 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42916.log.gz 2024-09-16T13:50:39.304 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109990.log.gz 2024-09-16T13:50:39.304 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.97895.log.gz 2024-09-16T13:50:39.304 INFO:teuthology.orchestra.run.smithi172.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.35099.log.gz 2024-09-16T13:50:39.304 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67684.log.gz 2024-09-16T13:50:39.304 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.48717.log.gz 2024-09-16T13:50:39.304 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104176.log.gz 2024-09-16T13:50:39.305 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113470.log.gz 2024-09-16T13:50:39.305 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.117116.log.gz 2024-09-16T13:50:39.305 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.117157.log.gz 2024-09-16T13:50:39.305 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104888.log.gz 2024-09-16T13:50:39.305 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.61062.log.gz 2024-09-16T13:50:39.305 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.49065.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.106869.log.gz 2024-09-16T13:50:39.305 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.305 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123764.log.gz 2024-09-16T13:50:39.305 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50252.log.gz 2024-09-16T13:50:39.305 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.34973.log.gz 2024-09-16T13:50:39.305 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110421.log.gz 2024-09-16T13:50:39.305 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57536.log.gz 2024-09-16T13:50:39.305 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135693.log.gz 2024-09-16T13:50:39.305 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123219.log.gz 2024-09-16T13:50:39.306 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90432.log.gz 2024-09-16T13:50:39.306 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131667.log.gz 2024-09-16T13:50:39.306 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.60199.log.gz 2024-09-16T13:50:39.306 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.60307.log.gz 2024-09-16T13:50:39.306 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66447.log.gz 2024-09-16T13:50:39.306 INFO:teuthology.orchestra.run.smithi172.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110340.log.gz 2024-09-16T13:50:39.306 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115991.log.gz 2024-09-16T13:50:39.306 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111932.log.gz 2024-09-16T13:50:39.306 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127776.log.gz 2024-09-16T13:50:39.306 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.110951.log.gz 2024-09-16T13:50:39.306 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81457.log.gz 2024-09-16T13:50:39.306 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110053.log.gz 2024-09-16T13:50:39.306 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57849.log.gz 2024-09-16T13:50:39.306 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37460.log.gz 2024-09-16T13:50:39.307 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39752.log.gz 2024-09-16T13:50:39.307 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43126.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.83770.log.gz 2024-09-16T13:50:39.307 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.307 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66362.log.gz 2024-09-16T13:50:39.307 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.95523.log.gz 2024-09-16T13:50:39.307 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.132896.log.gz 2024-09-16T13:50:39.307 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.133044.log.gz 2024-09-16T13:50:39.307 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.133084.log.gz 2024-09-16T13:50:39.307 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122940.log.gz 2024-09-16T13:50:39.307 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92493.log.gz 2024-09-16T13:50:39.307 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66028.log.gz 2024-09-16T13:50:39.307 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105165.log.gz 2024-09-16T13:50:39.307 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91904.log.gz 2024-09-16T13:50:39.307 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42544.log.gz 89.9% 2024-09-16T13:50:39.308 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.126454.log.gz 2024-09-16T13:50:39.308 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127341.log.gz 2024-09-16T13:50:39.308 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127549.log.gz 2024-09-16T13:50:39.308 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.126875.log.gz 2024-09-16T13:50:39.308 INFO:teuthology.orchestra.run.smithi172.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.38361.log.gz 2024-09-16T13:50:39.308 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.50574.log.gz 2024-09-16T13:50:39.308 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56544.log.gz 2024-09-16T13:50:39.308 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.61076.log.gz 2024-09-16T13:50:39.308 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49664.log.gz 2024-09-16T13:50:39.308 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.59988.log.gz 2024-09-16T13:50:39.308 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57746.log.gz 2024-09-16T13:50:39.308 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30640.log.gz 2024-09-16T13:50:39.308 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.36291.log.gz 2024-09-16T13:50:39.308 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55642.log.gz 2024-09-16T13:50:39.309 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54591.log.gz 2024-09-16T13:50:39.309 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.110402.log.gz 2024-09-16T13:50:39.309 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69896.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.117053.log.gz 2024-09-16T13:50:39.309 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.309 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123427.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.87532.log.gz 2024-09-16T13:50:39.309 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.309 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.87688.log.gz 2024-09-16T13:50:39.309 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.124690.log.gz 2024-09-16T13:50:39.309 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113612.log.gz 2024-09-16T13:50:39.309 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122540.log.gz 2024-09-16T13:50:39.309 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.36618.log.gz 2024-09-16T13:50:39.309 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.110207.log.gz 2024-09-16T13:50:39.309 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113027.log.gz 2024-09-16T13:50:39.310 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.86012.log.gz 2024-09-16T13:50:39.310 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52677.log.gz 2024-09-16T13:50:39.310 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41803.log.gz 2024-09-16T13:50:39.310 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66108.log.gz 2024-09-16T13:50:39.310 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53184.log.gz 2024-09-16T13:50:39.310 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% 89.8% -- replaced with /var/log/ceph/cluster1-client.admin.71271.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.56460.log.gz 2024-09-16T13:50:39.310 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.310 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123301.log.gz 2024-09-16T13:50:39.310 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90475.log.gz 2024-09-16T13:50:39.310 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80404.log.gz 2024-09-16T13:50:39.310 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.141417.log.gz 2024-09-16T13:50:39.310 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.141411.log.gz 2024-09-16T13:50:39.310 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98675.log.gz 2024-09-16T13:50:39.311 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.112417.log.gz 2024-09-16T13:50:39.311 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136911.log.gz 2024-09-16T13:50:39.311 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82313.log.gz 2024-09-16T13:50:39.311 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42496.log.gz 2024-09-16T13:50:39.311 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.100327.log.gz 2024-09-16T13:50:39.311 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.47892.log.gz 2024-09-16T13:50:39.311 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46181.log.gz 2024-09-16T13:50:39.311 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51727.log.gz 2024-09-16T13:50:39.311 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58083.log.gz 88.9% 2024-09-16T13:50:39.311 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.87975.log.gz 2024-09-16T13:50:39.311 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.88131.log.gz 2024-09-16T13:50:39.311 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.110161.log.gz 2024-09-16T13:50:39.311 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57535.log.gz 2024-09-16T13:50:39.311 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60713.log.gz 2024-09-16T13:50:39.312 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77834.log.gz 2024-09-16T13:50:39.312 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123136.log.gz 2024-09-16T13:50:39.312 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125949.log.gz 2024-09-16T13:50:39.312 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% 89.9% -- replaced with /var/log/ceph/cluster1-client.admin.129534.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.112208.log.gz 2024-09-16T13:50:39.312 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.312 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76988.log.gz 2024-09-16T13:50:39.312 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66658.log.gz 2024-09-16T13:50:39.312 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33160.log.gz 2024-09-16T13:50:39.312 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42542.log.gz 2024-09-16T13:50:39.312 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32240.log.gz 2024-09-16T13:50:39.312 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.80378.log.gz 2024-09-16T13:50:39.312 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.83838.log.gz 2024-09-16T13:50:39.312 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113402.log.gz 2024-09-16T13:50:39.312 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105376.log.gz 2024-09-16T13:50:39.313 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104994.log.gz 2024-09-16T13:50:39.313 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.108516.log.gz 2024-09-16T13:50:39.313 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78964.log.gz 2024-09-16T13:50:39.313 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.107479.log.gz 2024-09-16T13:50:39.313 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87779.log.gz 2024-09-16T13:50:39.313 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100401.log.gz 2024-09-16T13:50:39.313 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81140.log.gz 2024-09-16T13:50:39.313 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31189.log.gz 87.3% 2024-09-16T13:50:39.313 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.97175.log.gz 2024-09-16T13:50:39.313 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136743.log.gz 2024-09-16T13:50:39.313 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116750.log.gz 2024-09-16T13:50:39.313 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.141937.log.gz 2024-09-16T13:50:39.313 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.130057.log.gz 2024-09-16T13:50:39.314 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135968.log.gz 2024-09-16T13:50:39.314 INFO:teuthology.orchestra.run.smithi172.stderr: 66.1% -- replaced with /var/log/ceph/cluster1-client.mirror.38475.log.gz 2024-09-16T13:50:39.314 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131439.log.gz 2024-09-16T13:50:39.314 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112100.log.gz 2024-09-16T13:50:39.314 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112418.log.gz 2024-09-16T13:50:39.314 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42330.log.gz 2024-09-16T13:50:39.314 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34682.log.gz 2024-09-16T13:50:39.314 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55496.log.gz 2024-09-16T13:50:39.314 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67159.log.gz 2024-09-16T13:50:39.314 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64651.log.gz 2024-09-16T13:50:39.314 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98565.log.gz 2024-09-16T13:50:39.314 INFO:teuthology.orchestra.run.smithi172.stderr: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39159.log.gz 2024-09-16T13:50:39.314 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104868.log.gz 2024-09-16T13:50:39.315 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63122.log.gz 2024-09-16T13:50:39.315 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101244.log.gz 2024-09-16T13:50:39.315 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103059.log.gz 2024-09-16T13:50:39.315 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109541.log.gz 2024-09-16T13:50:39.315 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91187.log.gz 2024-09-16T13:50:39.315 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115294.log.gz 2024-09-16T13:50:39.315 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.107247.log.gz 2024-09-16T13:50:39.315 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93842.log.gz 2024-09-16T13:50:39.315 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.141413.log.gz 2024-09-16T13:50:39.315 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.141870.log.gz 2024-09-16T13:50:39.315 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122626.log.gz 2024-09-16T13:50:39.315 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.119512.log.gz 2024-09-16T13:50:39.315 INFO:teuthology.orchestra.run.smithi172.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.132144.log.gz 2024-09-16T13:50:39.315 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79242.log.gz 2024-09-16T13:50:39.316 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92832.log.gz 2024-09-16T13:50:39.316 INFO:teuthology.orchestra.run.smithi172.stderr: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.131553.log.gz 2024-09-16T13:50:39.316 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32661.log.gz 2024-09-16T13:50:39.316 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42331.log.gz 2024-09-16T13:50:39.316 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53185.log.gz 2024-09-16T13:50:39.316 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52367.log.gz 2024-09-16T13:50:39.316 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56126.log.gz 2024-09-16T13:50:39.316 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46073.log.gz 2024-09-16T13:50:39.316 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78362.log.gz 2024-09-16T13:50:39.316 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78613.log.gz 89.0% 2024-09-16T13:50:39.316 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.98419.log.gz 2024-09-16T13:50:39.316 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99647.log.gz 2024-09-16T13:50:39.316 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101019.log.gz 2024-09-16T13:50:39.317 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70005.log.gz 2024-09-16T13:50:39.317 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70132.log.gz 2024-09-16T13:50:39.317 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91441.log.gz 2024-09-16T13:50:39.317 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91860.log.gz 2024-09-16T13:50:39.317 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87270.log.gz 2024-09-16T13:50:39.317 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.95525.log.gz 2024-09-16T13:50:39.317 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91104.log.gz 2024-09-16T13:50:39.317 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108361.log.gz 2024-09-16T13:50:39.317 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91103.log.gz 2024-09-16T13:50:39.317 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123556.log.gz 2024-09-16T13:50:39.317 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster1-client.admin.111017.log.gz 2024-09-16T13:50:39.317 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111890.log.gz 2024-09-16T13:50:39.317 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.112731.log.gz 2024-09-16T13:50:39.317 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.131735.log.gz 2024-09-16T13:50:39.318 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139225.log.gz 2024-09-16T13:50:39.318 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32149.log.gz 89.9% 2024-09-16T13:50:39.318 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.34686.log.gz 2024-09-16T13:50:39.318 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134497.log.gz 2024-09-16T13:50:39.318 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31729.log.gz 2024-09-16T13:50:39.318 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.96906.log.gz 2024-09-16T13:50:39.318 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41655.log.gz 2024-09-16T13:50:39.318 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43336.log.gz 2024-09-16T13:50:39.318 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64460.log.gz 2024-09-16T13:50:39.318 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67078.log.gz 2024-09-16T13:50:39.318 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56964.log.gz 2024-09-16T13:50:39.318 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123766.log.gz 2024-09-16T13:50:39.318 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86313.log.gz 2024-09-16T13:50:39.318 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41190.log.gz 2024-09-16T13:50:39.318 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.92281.log.gz 2024-09-16T13:50:39.319 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105481.log.gz 2024-09-16T13:50:39.319 INFO:teuthology.orchestra.run.smithi172.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.131849.log.gz 2024-09-16T13:50:39.319 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100181.log.gz 2024-09-16T13:50:39.319 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.27677.log.gz 2024-09-16T13:50:39.319 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.27928.log.gz 2024-09-16T13:50:39.319 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.28057.log.gz 2024-09-16T13:50:39.319 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.28416.log.gz 2024-09-16T13:50:39.319 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.72552.log.gz 2024-09-16T13:50:39.319 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.109200.log.gz 2024-09-16T13:50:39.319 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88040.log.gz 2024-09-16T13:50:39.319 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112353.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.37068.log.gz 2024-09-16T13:50:39.319 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.319 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34135.log.gz 2024-09-16T13:50:39.319 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56401.log.gz 2024-09-16T13:50:39.320 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62648.log.gz 2024-09-16T13:50:39.320 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38091.log.gz 2024-09-16T13:50:39.320 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.141095.log.gz 2024-09-16T13:50:39.320 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster1-client.admin.133994.log.gz 2024-09-16T13:50:39.320 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.110972.log.gz 2024-09-16T13:50:39.320 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.128277.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.99363.log.gz 2024-09-16T13:50:39.320 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.320 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91061.log.gz 2024-09-16T13:50:39.320 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.36830.log.gz 2024-09-16T13:50:39.320 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.34633.log.gz 2024-09-16T13:50:39.320 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.129399.log.gz 2024-09-16T13:50:39.320 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30350.log.gz 2024-09-16T13:50:39.320 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34555.log.gz 2024-09-16T13:50:39.321 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.38179.log.gz 2024-09-16T13:50:39.321 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43965.log.gz 2024-09-16T13:50:39.321 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58269.log.gz 2024-09-16T13:50:39.321 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63678.log.gz 2024-09-16T13:50:39.321 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43272.log.gz 2024-09-16T13:50:39.321 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67832.log.gz 2024-09-16T13:50:39.321 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66824.log.gz 2024-09-16T13:50:39.321 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.98597.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.71738.log.gz 2024-09-16T13:50:39.321 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.321 INFO:teuthology.orchestra.run.smithi172.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.97684.log.gz 2024-09-16T13:50:39.321 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102851.log.gz 2024-09-16T13:50:39.321 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100536.log.gz 2024-09-16T13:50:39.321 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.100072.log.gz 2024-09-16T13:50:39.321 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.51173.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.90641.log.gz 2024-09-16T13:50:39.322 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.322 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135339.log.gz 2024-09-16T13:50:39.322 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122881.log.gz 2024-09-16T13:50:39.322 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123512.log.gz 2024-09-16T13:50:39.322 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112253.log.gz 2024-09-16T13:50:39.322 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126286.log.gz 2024-09-16T13:50:39.322 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.142542.log.gz 2024-09-16T13:50:39.322 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135086.log.gz 2024-09-16T13:50:39.322 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127276.log.gz 2024-09-16T13:50:39.322 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41254.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.134189.log.gz 2024-09-16T13:50:39.322 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.322 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122024.log.gz 2024-09-16T13:50:39.322 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55011.log.gz 2024-09-16T13:50:39.323 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63038.log.gz 2024-09-16T13:50:39.323 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66867.log.gz 2024-09-16T13:50:39.323 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.72722.log.gz 2024-09-16T13:50:39.323 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75636.log.gz 2024-09-16T13:50:39.323 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76842.log.gz 2024-09-16T13:50:39.323 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77771.log.gz 2024-09-16T13:50:39.323 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79410.log.gz 2024-09-16T13:50:39.323 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81518.log.gz 2024-09-16T13:50:39.323 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103310.log.gz 2024-09-16T13:50:39.323 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89508.log.gz 2024-09-16T13:50:39.323 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55284.log.gz 2024-09-16T13:50:39.323 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78191.log.gz 2024-09-16T13:50:39.323 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114494.log.gz 2024-09-16T13:50:39.323 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.119740.log.gz 2024-09-16T13:50:39.324 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138382.log.gz 2024-09-16T13:50:39.324 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123404.log.gz 2024-09-16T13:50:39.324 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126919.log.gz 2024-09-16T13:50:39.324 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.129404.log.gz 2024-09-16T13:50:39.324 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.36070.log.gz 2024-09-16T13:50:39.324 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30430.log.gz 2024-09-16T13:50:39.324 INFO:teuthology.orchestra.run.smithi172.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-client.admin.97544.log.gz 2024-09-16T13:50:39.324 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.131252.log.gz 2024-09-16T13:50:39.324 INFO:teuthology.orchestra.run.smithi172.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50118.log.gz 2024-09-16T13:50:39.324 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44385.log.gz 2024-09-16T13:50:39.324 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57389.log.gz 2024-09-16T13:50:39.324 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57114.log.gz 2024-09-16T13:50:39.324 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77226.log.gz 2024-09-16T13:50:39.325 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99624.log.gz 2024-09-16T13:50:39.325 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91062.log.gz 2024-09-16T13:50:39.325 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114517.log.gz 2024-09-16T13:50:39.325 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.118327.log.gz 2024-09-16T13:50:39.325 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.142190.log.gz 2024-09-16T13:50:39.325 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136347.log.gz 2024-09-16T13:50:39.325 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134242.log.gz 2024-09-16T13:50:39.325 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.135652.log.gz 2024-09-16T13:50:39.325 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.118310.log.gz 2024-09-16T13:50:39.325 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101122.log.gz 2024-09-16T13:50:39.325 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126243.log.gz 2024-09-16T13:50:39.325 INFO:teuthology.orchestra.run.smithi172.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.74061.log.gz 2024-09-16T13:50:39.325 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.95798.log.gz 2024-09-16T13:50:39.325 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113796.log.gz 2024-09-16T13:50:39.326 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72950.log.gz 2024-09-16T13:50:39.326 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.110857.log.gz 2024-09-16T13:50:39.326 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41697.log.gz 2024-09-16T13:50:39.326 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.74195.log.gz 2024-09-16T13:50:39.326 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.95527.log.gz 2024-09-16T13:50:39.326 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122967.log.gz 2024-09-16T13:50:39.326 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% 80.2% -- replaced with /var/log/ceph/cluster2-client.admin.118197.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.133450.log.gz 2024-09-16T13:50:39.326 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.326 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.73967.log.gz 2024-09-16T13:50:39.326 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100811.log.gz 2024-09-16T13:50:39.326 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.88018.log.gz 2024-09-16T13:50:39.326 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74134.log.gz 2024-09-16T13:50:39.326 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113654.log.gz 2024-09-16T13:50:39.326 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99841.log.gz 2024-09-16T13:50:39.327 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91020.log.gz 2024-09-16T13:50:39.327 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.76138.log.gz 2024-09-16T13:50:39.327 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137019.log.gz 2024-09-16T13:50:39.327 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70195.log.gz 2024-09-16T13:50:39.327 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92746.log.gz 2024-09-16T13:50:39.327 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93631.log.gz 2024-09-16T13:50:39.327 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89970.log.gz 2024-09-16T13:50:39.327 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39958.log.gz 2024-09-16T13:50:39.327 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89843.log.gz 2024-09-16T13:50:39.327 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55349.log.gz 2024-09-16T13:50:39.327 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.71275.log.gz 2024-09-16T13:50:39.327 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58502.log.gz 2024-09-16T13:50:39.327 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40534.log.gz 2024-09-16T13:50:39.327 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94437.log.gz 2024-09-16T13:50:39.328 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.94938.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.79561.log.gz 2024-09-16T13:50:39.328 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.328 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.89760.log.gz 2024-09-16T13:50:39.328 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138111.log.gz 2024-09-16T13:50:39.328 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139055.log.gz 2024-09-16T13:50:39.328 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108683.log.gz 2024-09-16T13:50:39.328 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78721.log.gz 2024-09-16T13:50:39.328 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.120981.log.gz 2024-09-16T13:50:39.328 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.117982.log.gz 2024-09-16T13:50:39.328 INFO:teuthology.orchestra.run.smithi172.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.120082.log.gz 2024-09-16T13:50:39.328 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42183.log.gz 2024-09-16T13:50:39.328 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120176.log.gz 2024-09-16T13:50:39.328 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.108858.log.gz 2024-09-16T13:50:39.329 INFO:teuthology.orchestra.run.smithi172.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.108838.log.gz 18.6% 2024-09-16T13:50:39.329 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.37547.log.gz 2024-09-16T13:50:39.329 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.141251.log.gz 2024-09-16T13:50:39.329 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100912.log.gz 2024-09-16T13:50:39.329 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99241.log.gz 2024-09-16T13:50:39.329 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46328.log.gz 2024-09-16T13:50:39.329 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101755.log.gz 2024-09-16T13:50:39.329 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124479.log.gz 2024-09-16T13:50:39.329 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103568.log.gz 2024-09-16T13:50:39.329 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40840.log.gz 2024-09-16T13:50:39.329 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56691.log.gz 2024-09-16T13:50:39.329 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101630.log.gz 2024-09-16T13:50:39.329 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.48117.log.gz 2024-09-16T13:50:39.329 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89130.log.gz 2024-09-16T13:50:39.330 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32870.log.gz 2024-09-16T13:50:39.330 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.116499.log.gz 2024-09-16T13:50:39.330 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72941.log.gz 2024-09-16T13:50:39.330 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.96271.log.gz 2024-09-16T13:50:39.330 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29337.log.gz 2024-09-16T13:50:39.330 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65820.log.gz 2024-09-16T13:50:39.330 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61582.log.gz 2024-09-16T13:50:39.330 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101547.log.gz 2024-09-16T13:50:39.330 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67494.log.gz 2024-09-16T13:50:39.330 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103126.log.gz 2024-09-16T13:50:39.330 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138447.log.gz 2024-09-16T13:50:39.330 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65312.log.gz 2024-09-16T13:50:39.330 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.109831.log.gz 2024-09-16T13:50:39.331 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33832.log.gz 2024-09-16T13:50:39.331 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45169.log.gz 2024-09-16T13:50:39.331 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65898.log.gz 2024-09-16T13:50:39.331 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45652.log.gz 2024-09-16T13:50:39.331 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109869.log.gz 2024-09-16T13:50:39.331 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101799.log.gz 2024-09-16T13:50:39.331 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.109676.log.gz 2024-09-16T13:50:39.331 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33922.log.gz 2024-09-16T13:50:39.331 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74573.log.gz 2024-09-16T13:50:39.331 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135021.log.gz 2024-09-16T13:50:39.331 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65615.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.32992.log.gz 2024-09-16T13:50:39.331 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.331 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.62535.log.gz 2024-09-16T13:50:39.331 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.59547.log.gz 2024-09-16T13:50:39.331 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.74867.log.gz 2024-09-16T13:50:39.332 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30893.log.gz 2024-09-16T13:50:39.332 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120404.log.gz 2024-09-16T13:50:39.332 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.62307.log.gz 2024-09-16T13:50:39.332 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101757.log.gz 2024-09-16T13:50:39.332 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33925.log.gz 2024-09-16T13:50:39.332 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65400.log.gz 2024-09-16T13:50:39.332 INFO:teuthology.orchestra.run.smithi172.stderr: 67.3% -- replaced with /var/log/ceph/cluster1-client.mirror.86126.log.gz 2024-09-16T13:50:39.332 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.133001.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.74476.log.gz 2024-09-16T13:50:39.332 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.332 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67498.log.gz 2024-09-16T13:50:39.332 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32871.log.gz 2024-09-16T13:50:39.332 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33623.log.gz 2024-09-16T13:50:39.333 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.admin.74336.log.gz 2024-09-16T13:50:39.333 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.71819.log.gz 2024-09-16T13:50:39.333 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46866.log.gz 2024-09-16T13:50:39.333 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101652.log.gz 2024-09-16T13:50:39.333 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.74362.log.gz 2024-09-16T13:50:39.333 INFO:teuthology.orchestra.run.smithi172.stderr: 77.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51976.log.gz 2024-09-16T13:50:39.333 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.96283.log.gz 2024-09-16T13:50:39.333 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30054.log.gz 2024-09-16T13:50:39.333 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104114.log.gz 2024-09-16T13:50:39.333 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65687.log.gz 2024-09-16T13:50:39.333 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33504.log.gz 2024-09-16T13:50:39.333 INFO:teuthology.orchestra.run.smithi172.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.132258.log.gz 2024-09-16T13:50:39.333 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113760.log.gz 2024-09-16T13:50:39.333 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76295.log.gz 2024-09-16T13:50:39.334 INFO:teuthology.orchestra.run.smithi172.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.62246.log.gz 90.0% 2024-09-16T13:50:39.334 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.30772.log.gz 2024-09-16T13:50:39.334 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.107755.log.gz 2024-09-16T13:50:39.334 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.120554.log.gz 2024-09-16T13:50:39.334 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124560.log.gz 2024-09-16T13:50:39.334 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88531.log.gz 2024-09-16T13:50:39.334 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.74818.log.gz 2024-09-16T13:50:39.334 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.76376.log.gz 2024-09-16T13:50:39.334 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.admin.120337.log.gz 2024-09-16T13:50:39.334 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33502.log.gz 2024-09-16T13:50:39.334 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32779.log.gz 2024-09-16T13:50:39.334 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120518.log.gz 2024-09-16T13:50:39.334 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115780.log.gz 2024-09-16T13:50:39.334 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45862.log.gz 2024-09-16T13:50:39.335 INFO:teuthology.orchestra.run.smithi172.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.99432.log.gz 2024-09-16T13:50:39.335 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67367.log.gz 2024-09-16T13:50:39.335 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111344.log.gz 2024-09-16T13:50:39.335 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.33624.log.gz 2024-09-16T13:50:39.335 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.129771.log.gz 2024-09-16T13:50:39.335 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93125.log.gz 2024-09-16T13:50:39.335 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124539.log.gz 2024-09-16T13:50:39.335 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.130899.log.gz 2024-09-16T13:50:39.335 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40978.log.gz 2024-09-16T13:50:39.336 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138531.log.gz 2024-09-16T13:50:39.336 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56902.log.gz 2024-09-16T13:50:39.336 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54725.log.gz 2024-09-16T13:50:39.336 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101020.log.gz 2024-09-16T13:50:39.336 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109738.log.gz 2024-09-16T13:50:39.336 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124688.log.gz 2024-09-16T13:50:39.336 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60766.log.gz 2024-09-16T13:50:39.336 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114770.log.gz 2024-09-16T13:50:39.336 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.59724.log.gz 2024-09-16T13:50:39.336 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115570.log.gz 2024-09-16T13:50:39.336 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88448.log.gz 2024-09-16T13:50:39.336 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29016.log.gz 2024-09-16T13:50:39.336 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40811.log.gz 2024-09-16T13:50:39.337 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54089.log.gz 2024-09-16T13:50:39.337 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76519.log.gz 2024-09-16T13:50:39.337 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60776.log.gz 2024-09-16T13:50:39.337 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40979.log.gz 2024-09-16T13:50:39.337 INFO:teuthology.orchestra.run.smithi172.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.admin.120451.log.gz 2024-09-16T13:50:39.337 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86447.log.gz 2024-09-16T13:50:39.337 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88415.log.gz 2024-09-16T13:50:39.337 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29015.log.gz 2024-09-16T13:50:39.337 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101655.log.gz 2024-09-16T13:50:39.337 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53961.log.gz 2024-09-16T13:50:39.337 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104175.log.gz 2024-09-16T13:50:39.338 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61904.log.gz 2024-09-16T13:50:39.338 INFO:teuthology.orchestra.run.smithi172.stderr: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.26955.log.gz 2024-09-16T13:50:39.338 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.76465.log.gz 2024-09-16T13:50:39.338 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81669.log.gz 2024-09-16T13:50:39.338 INFO:teuthology.orchestra.run.smithi172.stderr: 88.8% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45018.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.88332.log.gz 2024-09-16T13:50:39.338 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.338 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54171.log.gz 2024-09-16T13:50:39.338 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89216.log.gz 2024-09-16T13:50:39.338 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98106.log.gz 2024-09-16T13:50:39.338 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45168.log.gz 2024-09-16T13:50:39.338 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77730.log.gz 2024-09-16T13:50:39.338 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89006.log.gz 2024-09-16T13:50:39.338 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88961.log.gz 2024-09-16T13:50:39.339 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102574.log.gz 2024-09-16T13:50:39.339 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.76378.log.gz 2024-09-16T13:50:39.339 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68634.log.gz 2024-09-16T13:50:39.339 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98992.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.109896.log.gz 2024-09-16T13:50:39.339 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.339 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56822.log.gz 2024-09-16T13:50:39.339 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137691.log.gz 2024-09-16T13:50:39.339 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.111563.log.gz 2024-09-16T13:50:39.339 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68551.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.68782.log.gz 2024-09-16T13:50:39.339 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.339 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101656.log.gz 2024-09-16T13:50:39.339 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101545.log.gz 2024-09-16T13:50:39.339 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46494.log.gz 89.9% 2024-09-16T13:50:39.339 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.69183.log.gz 2024-09-16T13:50:39.340 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41025.log.gz 2024-09-16T13:50:39.340 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134962.log.gz 2024-09-16T13:50:39.340 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93422.log.gz 2024-09-16T13:50:39.340 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110653.log.gz 2024-09-16T13:50:39.340 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125024.log.gz 2024-09-16T13:50:39.340 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56693.log.gz 2024-09-16T13:50:39.340 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.83769.log.gz 2024-09-16T13:50:39.340 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103337.log.gz 2024-09-16T13:50:39.340 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.48095.log.gz 2024-09-16T13:50:39.340 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81411.log.gz 2024-09-16T13:50:39.340 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.62421.log.gz 2024-09-16T13:50:39.340 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40916.log.gz 2024-09-16T13:50:39.340 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102596.log.gz 2024-09-16T13:50:39.340 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93464.log.gz 2024-09-16T13:50:39.341 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103857.log.gz 2024-09-16T13:50:39.341 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88490.log.gz 2024-09-16T13:50:39.341 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.141789.log.gz 2024-09-16T13:50:39.341 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37352.log.gz 2024-09-16T13:50:39.341 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74782.log.gz 2024-09-16T13:50:39.341 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63193.log.gz 2024-09-16T13:50:39.341 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93084.log.gz 2024-09-16T13:50:39.341 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.39845.log.gz 2024-09-16T13:50:39.341 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.128982.log.gz 2024-09-16T13:50:39.341 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111428.log.gz 2024-09-16T13:50:39.341 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.134879.log.gz 2024-09-16T13:50:39.341 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93546.log.gz 2024-09-16T13:50:39.341 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108911.log.gz 2024-09-16T13:50:39.341 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105289.log.gz 2024-09-16T13:50:39.342 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64892.log.gz 2024-09-16T13:50:39.342 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.30220.log.gz 2024-09-16T13:50:39.342 INFO:teuthology.orchestra.run.smithi172.stderr: 19.0% 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.120597.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.74309.log.gz 2024-09-16T13:50:39.342 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.342 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.109712.log.gz 2024-09-16T13:50:39.342 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102785.log.gz 2024-09-16T13:50:39.342 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68891.log.gz 2024-09-16T13:50:39.342 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.72338.log.gz 2024-09-16T13:50:39.342 INFO:teuthology.orchestra.run.smithi172.stderr: 80.0% 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.74647.log.gz 2024-09-16T13:50:39.342 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.60940.log.gz 2024-09-16T13:50:39.342 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61017.log.gz 2024-09-16T13:50:39.342 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.76355.log.gz 2024-09-16T13:50:39.342 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105119.log.gz 2024-09-16T13:50:39.343 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125950.log.gz 2024-09-16T13:50:39.343 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113822.log.gz 2024-09-16T13:50:39.343 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100411.log.gz 2024-09-16T13:50:39.343 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134751.log.gz 2024-09-16T13:50:39.343 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124269.log.gz 2024-09-16T13:50:39.343 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.59773.log.gz 2024-09-16T13:50:39.343 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62515.log.gz 2024-09-16T13:50:39.343 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81668.log.gz 2024-09-16T13:50:39.343 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67496.log.gz 2024-09-16T13:50:39.343 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.admin.62448.log.gz 2024-09-16T13:50:39.343 INFO:teuthology.orchestra.run.smithi172.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73564.log.gz 2024-09-16T13:50:39.343 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138955.log.gz 2024-09-16T13:50:39.343 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51882.log.gz 2024-09-16T13:50:39.343 INFO:teuthology.orchestra.run.smithi172.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58756.log.gz 2024-09-16T13:50:39.344 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69100.log.gz 2024-09-16T13:50:39.344 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.76218.log.gz 2024-09-16T13:50:39.344 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37253.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.50822.log.gz 2024-09-16T13:50:39.344 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.344 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.96124.log.gz 2024-09-16T13:50:39.344 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.59769.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.120363.log.gz 2024-09-16T13:50:39.344 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.344 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.74781.log.gz 2024-09-16T13:50:39.344 INFO:teuthology.orchestra.run.smithi172.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.118838.log.gz 2024-09-16T13:50:39.344 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52705.log.gz 2024-09-16T13:50:39.344 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29057.log.gz 2024-09-16T13:50:39.344 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30771.log.gz 2024-09-16T13:50:39.344 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40031.log.gz 2024-09-16T13:50:39.345 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.52261.log.gz 2024-09-16T13:50:39.345 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.119398.log.gz 2024-09-16T13:50:39.345 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101243.log.gz 2024-09-16T13:50:39.345 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139163.log.gz 2024-09-16T13:50:39.345 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54510.log.gz 2024-09-16T13:50:39.345 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60786.log.gz 2024-09-16T13:50:39.345 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33291.log.gz 2024-09-16T13:50:39.345 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56903.log.gz 2024-09-16T13:50:39.345 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.30136.log.gz 2024-09-16T13:50:39.345 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29799.log.gz 2024-09-16T13:50:39.345 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76093.log.gz 2024-09-16T13:50:39.345 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131895.log.gz 2024-09-16T13:50:39.345 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124664.log.gz 2024-09-16T13:50:39.345 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56755.log.gz 2024-09-16T13:50:39.346 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80718.log.gz 2024-09-16T13:50:39.346 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102595.log.gz 2024-09-16T13:50:39.346 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68976.log.gz 2024-09-16T13:50:39.346 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.75544.log.gz 2024-09-16T13:50:39.346 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.29143.log.gz 2024-09-16T13:50:39.346 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.139014.log.gz 2024-09-16T13:50:39.346 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89090.log.gz 2024-09-16T13:50:39.346 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138592.log.gz 2024-09-16T13:50:39.346 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112041.log.gz 2024-09-16T13:50:39.346 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46332.log.gz 2024-09-16T13:50:39.346 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54656.log.gz 2024-09-16T13:50:39.346 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68680.log.gz 2024-09-16T13:50:39.346 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61945.log.gz 2024-09-16T13:50:39.347 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35100.log.gz 2024-09-16T13:50:39.347 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30053.log.gz 2024-09-16T13:50:39.347 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67289.log.gz 2024-09-16T13:50:39.347 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46221.log.gz 2024-09-16T13:50:39.347 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116415.log.gz 2024-09-16T13:50:39.347 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58419.log.gz 2024-09-16T13:50:39.347 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.109253.log.gz 2024-09-16T13:50:39.347 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.49435.log.gz 2024-09-16T13:50:39.347 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70477.log.gz 2024-09-16T13:50:39.347 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29758.log.gz 2024-09-16T13:50:39.347 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64238.log.gz 2024-09-16T13:50:39.347 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89928.log.gz 2024-09-16T13:50:39.347 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56880.log.gz 2024-09-16T13:50:39.347 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.120955.log.gz 2024-09-16T13:50:39.348 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118550.log.gz 2024-09-16T13:50:39.348 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138869.log.gz 2024-09-16T13:50:39.348 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98268.log.gz 2024-09-16T13:50:39.348 INFO:teuthology.orchestra.run.smithi172.stderr: 80.6% -- replaced with /var/log/ceph/cluster1-client.mirror.39556.log.gz 2024-09-16T13:50:39.348 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.31609.log.gz 2024-09-16T13:50:39.348 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.48120.log.gz 2024-09-16T13:50:39.348 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.37251.log.gz 2024-09-16T13:50:39.348 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.34214.log.gz 2024-09-16T13:50:39.348 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55704.log.gz 2024-09-16T13:50:39.348 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54720.log.gz 2024-09-16T13:50:39.349 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42705.log.gz 2024-09-16T13:50:39.349 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65732.log.gz 2024-09-16T13:50:39.349 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45701.log.gz 2024-09-16T13:50:39.349 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32529.log.gz 2024-09-16T13:50:39.349 INFO:teuthology.orchestra.run.smithi172.stderr: 80.5% 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.26748.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.71736.log.gz 2024-09-16T13:50:39.349 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.349 INFO:teuthology.orchestra.run.smithi172.stderr: 56.3% -- replaced with /var/log/ceph/cluster1-client.mirror.26884.log.gz 2024-09-16T13:50:39.349 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.27353.log.gz 2024-09-16T13:50:39.349 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75608.log.gz 2024-09-16T13:50:39.349 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105938.log.gz 2024-09-16T13:50:39.349 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% 89.5% -- replaced with /var/log/ceph/cluster2-client.admin.107831.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.107922.log.gz 2024-09-16T13:50:39.349 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.349 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.114873.log.gz 2024-09-16T13:50:39.350 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.133887.log.gz 2024-09-16T13:50:39.350 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.28846.log.gz 2024-09-16T13:50:39.350 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136138.log.gz 2024-09-16T13:50:39.350 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119699.log.gz 2024-09-16T13:50:39.350 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.135234.log.gz 2024-09-16T13:50:39.350 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122836.log.gz 2024-09-16T13:50:39.350 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127506.log.gz 2024-09-16T13:50:39.350 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105098.log.gz 2024-09-16T13:50:39.350 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.48116.log.gz 2024-09-16T13:50:39.350 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.72386.log.gz 2024-09-16T13:50:39.352 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30304.log.gz 2024-09-16T13:50:39.352 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.112101.log.gz 2024-09-16T13:50:39.352 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82222.log.gz 2024-09-16T13:50:39.352 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.87817.log.gz 2024-09-16T13:50:39.352 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79874.log.gz 2024-09-16T13:50:39.352 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116246.log.gz 2024-09-16T13:50:39.352 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115360.log.gz 2024-09-16T13:50:39.352 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115550.log.gz 2024-09-16T13:50:39.352 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112562.log.gz 2024-09-16T13:50:39.353 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139373.log.gz 2024-09-16T13:50:39.353 INFO:teuthology.orchestra.run.smithi172.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.120135.log.gz 2024-09-16T13:50:39.353 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.135169.log.gz 2024-09-16T13:50:39.353 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.121286.log.gz 2024-09-16T13:50:39.353 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117158.log.gz 2024-09-16T13:50:39.355 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44072.log.gz 2024-09-16T13:50:39.357 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45549.log.gz 2024-09-16T13:50:39.360 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110561.log.gz 2024-09-16T13:50:39.360 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.110376.log.gz 2024-09-16T13:50:39.369 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45800.log.gz 2024-09-16T13:50:39.372 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138659.log.gz 2024-09-16T13:50:39.377 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64738.log.gz 2024-09-16T13:50:39.381 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102430.log.gz 2024-09-16T13:50:39.414 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.86961.log.gz 2024-09-16T13:50:39.418 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34042.log.gz 2024-09-16T13:50:39.428 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42603.log.gz 2024-09-16T13:50:39.435 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43801.log.gz 2024-09-16T13:50:39.437 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.27727.log.gz 2024-09-16T13:50:39.440 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53250.log.gz 2024-09-16T13:50:39.444 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100958.log.gz 2024-09-16T13:50:39.449 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42961.log.gz 2024-09-16T13:50:39.453 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92535.log.gz 2024-09-16T13:50:39.643 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.75247.log.gz 2024-09-16T13:50:39.643 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.141606.log.gz 2024-09-16T13:50:39.643 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123555.log.gz 2024-09-16T13:50:39.644 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81519.log.gz 2024-09-16T13:50:39.646 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43966.log.gz 2024-09-16T13:50:39.649 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65057.log.gz 2024-09-16T13:50:39.651 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66528.log.gz 2024-09-16T13:50:39.656 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125236.log.gz 2024-09-16T13:50:39.657 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134579.log.gz 2024-09-16T13:50:39.659 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111604.log.gz 2024-09-16T13:50:39.662 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54360.log.gz 2024-09-16T13:50:39.664 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.49014.log.gz 2024-09-16T13:50:39.671 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120269.log.gz 2024-09-16T13:50:39.674 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79243.log.gz 2024-09-16T13:50:39.677 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92915.log.gz 2024-09-16T13:50:39.679 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.75087.log.gz 2024-09-16T13:50:39.682 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77897.log.gz 2024-09-16T13:50:39.684 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89215.log.gz 2024-09-16T13:50:39.688 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.84025.log.gz 2024-09-16T13:50:39.691 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.85743.log.gz 2024-09-16T13:50:39.692 INFO:teuthology.orchestra.run.smithi172.stderr: 19.2% -- replaced with /var/log/ceph/cluster1-client.admin.98210.log.gz 2024-09-16T13:50:39.701 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89844.log.gz 2024-09-16T13:50:39.705 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.141100.log.gz 2024-09-16T13:50:39.709 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103969.log.gz 2024-09-16T13:50:39.718 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45069.log.gz 2024-09-16T13:50:39.726 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67834.log.gz 2024-09-16T13:50:39.727 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67917.log.gz 2024-09-16T13:50:39.870 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% 90.1% -- replaced with /var/log/ceph/cluster1-client.admin.119653.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.140170.log.gz 2024-09-16T13:50:39.871 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.871 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102914.log.gz 2024-09-16T13:50:39.871 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.60514.log.gz 2024-09-16T13:50:39.871 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73926.log.gz 2024-09-16T13:50:39.871 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31309.log.gz 2024-09-16T13:50:39.871 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138028.log.gz 2024-09-16T13:50:39.871 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66992.log.gz 2024-09-16T13:50:39.871 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.128994.log.gz 2024-09-16T13:50:39.871 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134286.log.gz 2024-09-16T13:50:39.871 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31099.log.gz 2024-09-16T13:50:39.873 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 89.9% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124244.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.78091.log.gz 2024-09-16T13:50:39.873 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.133165.log.gz 2024-09-16T13:50:39.873 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.873 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52191.log.gz 2024-09-16T13:50:39.873 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103692.log.gz 2024-09-16T13:50:39.873 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92071.log.gz 2024-09-16T13:50:39.873 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66573.log.gz 2024-09-16T13:50:39.873 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.122095.log.gz 2024-09-16T13:50:39.873 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37353.log.gz 2024-09-16T13:50:39.874 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51575.log.gz 2024-09-16T13:50:39.874 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% 18.6% -- replaced with /var/log/ceph/cluster1-client.mirror.74999.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.72045.log.gz 2024-09-16T13:50:39.874 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.874 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70174.log.gz 2024-09-16T13:50:39.874 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster2-client.admin.49712.log.gz 2024-09-16T13:50:39.874 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50976.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.43174.log.gz 2024-09-16T13:50:39.874 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:39.874 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.71269.log.gz 2024-09-16T13:50:39.874 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128075.log.gz 2024-09-16T13:50:39.874 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109806.log.gz 2024-09-16T13:50:39.874 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114139.log.gz 2024-09-16T13:50:39.885 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68472.log.gz 2024-09-16T13:50:39.886 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68552.log.gz 2024-09-16T13:50:39.894 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46909.log.gz 2024-09-16T13:50:40.053 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.27485.log.gz 2024-09-16T13:50:40.079 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.141249.log.gz 2024-09-16T13:50:40.079 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120041.log.gz 2024-09-16T13:50:40.081 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.60505.log.gz 2024-09-16T13:50:40.081 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115295.log.gz 2024-09-16T13:50:40.086 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137121.log.gz 2024-09-16T13:50:40.087 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126920.log.gz 2024-09-16T13:50:40.089 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127085.log.gz 2024-09-16T13:50:40.095 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91859.log.gz 2024-09-16T13:50:40.097 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89592.log.gz 2024-09-16T13:50:40.098 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79202.log.gz 2024-09-16T13:50:40.101 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105228.log.gz 2024-09-16T13:50:40.103 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93001.log.gz 2024-09-16T13:50:40.109 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.108972.log.gz 2024-09-16T13:50:40.112 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90057.log.gz 2024-09-16T13:50:40.116 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.28525.log.gz 2024-09-16T13:50:40.134 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44430.log.gz 2024-09-16T13:50:40.138 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37252.log.gz 2024-09-16T13:50:40.142 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.60918.log.gz 2024-09-16T13:50:40.151 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100182.log.gz 2024-09-16T13:50:40.154 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33201.log.gz 2024-09-16T13:50:40.168 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94159.log.gz 2024-09-16T13:50:40.171 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113257.log.gz 2024-09-16T13:50:40.172 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125294.log.gz 2024-09-16T13:50:40.189 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.131191.log.gz 2024-09-16T13:50:40.190 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103967.log.gz 2024-09-16T13:50:40.198 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132590.log.gz 2024-09-16T13:50:40.199 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.141099.log.gz 2024-09-16T13:50:40.200 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123554.log.gz 2024-09-16T13:50:40.202 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138324.log.gz 2024-09-16T13:50:40.204 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125799.log.gz 2024-09-16T13:50:40.209 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126708.log.gz 2024-09-16T13:50:40.210 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134222.log.gz 2024-09-16T13:50:40.218 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.96086.log.gz 2024-09-16T13:50:40.218 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.73947.log.gz 2024-09-16T13:50:40.222 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70525.log.gz 2024-09-16T13:50:40.224 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67077.log.gz 2024-09-16T13:50:40.227 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124394.log.gz 2024-09-16T13:50:40.230 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75086.log.gz 2024-09-16T13:50:40.233 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76676.log.gz 2024-09-16T13:50:40.238 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78992.log.gz 2024-09-16T13:50:40.246 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100492.log.gz 2024-09-16T13:50:40.252 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.99772.log.gz 2024-09-16T13:50:40.254 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.98880.log.gz 2024-09-16T13:50:40.257 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91565.log.gz 2024-09-16T13:50:40.259 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81309.log.gz 2024-09-16T13:50:40.261 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81458.log.gz 2024-09-16T13:50:40.267 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56608.log.gz 2024-09-16T13:50:40.270 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63959.log.gz 2024-09-16T13:50:40.273 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63101.log.gz 2024-09-16T13:50:40.273 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66445.log.gz 2024-09-16T13:50:40.282 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65734.log.gz 2024-09-16T13:50:40.286 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.83840.log.gz 2024-09-16T13:50:40.291 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55223.log.gz 2024-09-16T13:50:40.296 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.135441.log.gz 2024-09-16T13:50:40.299 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101336.log.gz 2024-09-16T13:50:40.300 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86655.log.gz 2024-09-16T13:50:40.303 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.94937.log.gz 2024-09-16T13:50:40.313 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138003.log.gz 2024-09-16T13:50:40.326 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80043.log.gz 2024-09-16T13:50:40.333 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122882.log.gz 2024-09-16T13:50:40.354 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37230.log.gz 2024-09-16T13:50:40.372 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.36477.log.gz 2024-09-16T13:50:40.373 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41594.log.gz 2024-09-16T13:50:40.375 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43022.log.gz 2024-09-16T13:50:40.377 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44492.log.gz 2024-09-16T13:50:40.377 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45231.log.gz 2024-09-16T13:50:40.380 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.34344.log.gz 2024-09-16T13:50:40.384 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86333.log.gz 2024-09-16T13:50:40.387 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53668.log.gz 2024-09-16T13:50:40.388 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67833.log.gz 2024-09-16T13:50:40.389 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42813.log.gz 2024-09-16T13:50:40.389 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68044.log.gz 2024-09-16T13:50:40.391 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.72348.log.gz 2024-09-16T13:50:40.394 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34134.log.gz 2024-09-16T13:50:40.397 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44744.log.gz 2024-09-16T13:50:40.399 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44871.log.gz 2024-09-16T13:50:40.400 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.61066.log.gz 2024-09-16T13:50:40.405 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91482.log.gz 2024-09-16T13:50:40.407 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89717.log.gz 2024-09-16T13:50:40.413 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118611.log.gz 2024-09-16T13:50:40.417 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.119995.log.gz 2024-09-16T13:50:40.418 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.131279.log.gz 2024-09-16T13:50:40.418 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.131393.log.gz 2024-09-16T13:50:40.538 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.38038.log.gz 2024-09-16T13:50:40.581 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.48548.log.gz 2024-09-16T13:50:40.581 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.48520.log.gz 2024-09-16T13:50:40.583 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.37062.log.gz 2024-09-16T13:50:40.586 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30849.log.gz 2024-09-16T13:50:40.596 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48880.log.gz 2024-09-16T13:50:40.597 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57664.log.gz 2024-09-16T13:50:40.598 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57666.log.gz 2024-09-16T13:50:40.599 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.84889.log.gz 2024-09-16T13:50:40.601 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54150.log.gz 2024-09-16T13:50:40.602 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54173.log.gz 2024-09-16T13:50:40.606 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77774.log.gz 2024-09-16T13:50:40.606 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.73739.log.gz 2024-09-16T13:50:40.607 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127005.log.gz 2024-09-16T13:50:40.608 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80614.log.gz 2024-09-16T13:50:40.610 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.85337.log.gz 2024-09-16T13:50:40.613 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78361.log.gz 2024-09-16T13:50:40.614 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89635.log.gz 2024-09-16T13:50:40.615 INFO:teuthology.orchestra.run.smithi172.stderr: 91.0% -- replaced with /var/log/ceph/cluster1-client.admin.74222.log.gz 2024-09-16T13:50:40.618 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.110305.log.gz 2024-09-16T13:50:40.621 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114559.log.gz 2024-09-16T13:50:40.623 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.129940.log.gz 2024-09-16T13:50:40.625 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140004.log.gz 2024-09-16T13:50:40.626 INFO:teuthology.orchestra.run.smithi172.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.110909.log.gz 2024-09-16T13:50:40.630 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136767.log.gz 2024-09-16T13:50:40.632 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.130426.log.gz 2024-09-16T13:50:40.642 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.36072.log.gz 2024-09-16T13:50:40.651 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster2-client.admin.49138.log.gz 2024-09-16T13:50:40.659 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.87780.log.gz 2024-09-16T13:50:40.665 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34424.log.gz 2024-09-16T13:50:40.673 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50936.log.gz 2024-09-16T13:50:40.677 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30977.log.gz 2024-09-16T13:50:40.681 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123510.log.gz 2024-09-16T13:50:40.686 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34812.log.gz 2024-09-16T13:50:40.691 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31354.log.gz 2024-09-16T13:50:40.695 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54930.log.gz 2024-09-16T13:50:40.704 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46580.log.gz 2024-09-16T13:50:40.707 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99242.log.gz 2024-09-16T13:50:40.710 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57955.log.gz 2024-09-16T13:50:40.711 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.85670.log.gz 2024-09-16T13:50:40.711 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56165.log.gz 2024-09-16T13:50:40.714 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68889.log.gz 2024-09-16T13:50:40.718 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100837.log.gz 2024-09-16T13:50:40.721 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77042.log.gz 2024-09-16T13:50:40.724 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132388.log.gz 2024-09-16T13:50:40.726 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.98372.log.gz 2024-09-16T13:50:40.730 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111277.log.gz 2024-09-16T13:50:40.732 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112815.log.gz 2024-09-16T13:50:40.737 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113151.log.gz 2024-09-16T13:50:40.739 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.106920.log.gz 2024-09-16T13:50:40.741 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.109988.log.gz 2024-09-16T13:50:40.760 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137062.log.gz 2024-09-16T13:50:40.761 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137228.log.gz 2024-09-16T13:50:40.764 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139288.log.gz 2024-09-16T13:50:40.766 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138744.log.gz 2024-09-16T13:50:40.767 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138279.log.gz 2024-09-16T13:50:40.769 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126918.log.gz 2024-09-16T13:50:40.778 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.141871.log.gz 2024-09-16T13:50:40.780 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.125234.log.gz 2024-09-16T13:50:40.783 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122969.log.gz 2024-09-16T13:50:40.789 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103732.log.gz 2024-09-16T13:50:40.793 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.28247.log.gz 2024-09-16T13:50:40.793 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134110.log.gz 2024-09-16T13:50:40.793 INFO:teuthology.orchestra.run.smithi172.stderr: 19.2% -- replaced with /var/log/ceph/cluster1-client.admin.28439.log.gz 2024-09-16T13:50:40.802 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.36774.log.gz 2024-09-16T13:50:40.815 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45488.log.gz 2024-09-16T13:50:40.849 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43593.log.gz 2024-09-16T13:50:40.852 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57956.log.gz 2024-09-16T13:50:40.854 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52043.log.gz 2024-09-16T13:50:40.857 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69521.log.gz 2024-09-16T13:50:40.857 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61676.log.gz 2024-09-16T13:50:40.860 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63782.log.gz 2024-09-16T13:50:40.862 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67413.log.gz 2024-09-16T13:50:40.864 INFO:teuthology.orchestra.run.smithi172.stderr: 17.9% -- replaced with /var/log/ceph/cluster2-client.admin.108033.log.gz 2024-09-16T13:50:40.868 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103338.log.gz 2024-09-16T13:50:40.873 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79769.log.gz 2024-09-16T13:50:40.876 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.33503.log.gz 2024-09-16T13:50:40.876 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.118744.log.gz 2024-09-16T13:50:40.878 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136704.log.gz 2024-09-16T13:50:40.882 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.117981.log.gz 2024-09-16T13:50:40.886 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115083.log.gz 2024-09-16T13:50:40.892 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137021.log.gz 2024-09-16T13:50:40.895 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123279.log.gz 2024-09-16T13:50:40.898 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89695.log.gz 2024-09-16T13:50:40.901 INFO:teuthology.orchestra.run.smithi172.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.admin.97658.log.gz 2024-09-16T13:50:40.925 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138845.log.gz 2024-09-16T13:50:40.927 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.78825.log.gz 2024-09-16T13:50:40.936 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.40695.log.gz 2024-09-16T13:50:40.941 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70089.log.gz 2024-09-16T13:50:40.944 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92452.log.gz 2024-09-16T13:50:40.946 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42751.log.gz 2024-09-16T13:50:40.949 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46180.log.gz 2024-09-16T13:50:40.950 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62038.log.gz 2024-09-16T13:50:40.951 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41721.log.gz 2024-09-16T13:50:40.958 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42334.log.gz 2024-09-16T13:50:40.958 INFO:teuthology.orchestra.run.smithi172.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73906.log.gz 2024-09-16T13:50:40.961 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64847.log.gz 2024-09-16T13:50:40.962 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.97631.log.gz 2024-09-16T13:50:40.964 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.66948.log.gz 2024-09-16T13:50:40.971 INFO:teuthology.orchestra.run.smithi172.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.86468.log.gz 2024-09-16T13:50:40.974 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103143.log.gz 2024-09-16T13:50:40.976 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.107052.log.gz 2024-09-16T13:50:40.980 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90221.log.gz 2024-09-16T13:50:40.982 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92875.log.gz 2024-09-16T13:50:40.986 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.84288.log.gz 2024-09-16T13:50:40.988 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90852.log.gz 2024-09-16T13:50:40.991 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39843.log.gz 2024-09-16T13:50:41.000 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114181.log.gz 2024-09-16T13:50:41.001 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.118947.log.gz 2024-09-16T13:50:41.003 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138490.log.gz 2024-09-16T13:50:41.005 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138634.log.gz 2024-09-16T13:50:41.020 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.142009.log.gz 2024-09-16T13:50:41.022 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116580.log.gz 2024-09-16T13:50:41.022 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.131366.log.gz 2024-09-16T13:50:41.023 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.132237.log.gz 2024-09-16T13:50:41.025 INFO:teuthology.orchestra.run.smithi172.stderr: 90.7% -- replaced with /var/log/ceph/cluster1-client.admin.62334.log.gz 2024-09-16T13:50:41.030 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113970.log.gz 2024-09-16T13:50:41.039 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139290.log.gz 2024-09-16T13:50:41.063 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30979.log.gz 2024-09-16T13:50:41.070 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44703.log.gz 2024-09-16T13:50:41.073 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64441.log.gz 2024-09-16T13:50:41.079 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55029.log.gz 2024-09-16T13:50:41.088 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46011.log.gz 2024-09-16T13:50:41.090 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.59983.log.gz 2024-09-16T13:50:41.092 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30177.log.gz 2024-09-16T13:50:41.093 INFO:teuthology.orchestra.run.smithi172.stderr: 80.6% -- replaced with /var/log/ceph/cluster1-client.mirror.27316.log.gz 2024-09-16T13:50:41.095 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67920.log.gz 2024-09-16T13:50:41.119 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45799.log.gz 2024-09-16T13:50:41.121 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66154.log.gz 2024-09-16T13:50:41.126 INFO:teuthology.orchestra.run.smithi172.stderr: 17.4% -- replaced with /var/log/ceph/cluster1-client.admin.87907.log.gz 2024-09-16T13:50:41.126 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81984.log.gz 2024-09-16T13:50:41.129 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81459.log.gz 2024-09-16T13:50:41.129 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.92622.log.gz 2024-09-16T13:50:41.131 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92115.log.gz 2024-09-16T13:50:41.135 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94118.log.gz 2024-09-16T13:50:41.136 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77565.log.gz 2024-09-16T13:50:41.141 INFO:teuthology.orchestra.run.smithi172.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.110123.log.gz 2024-09-16T13:50:41.142 INFO:teuthology.orchestra.run.smithi172.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86427.log.gz 2024-09-16T13:50:41.143 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.117804.log.gz 2024-09-16T13:50:41.145 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102303.log.gz 2024-09-16T13:50:41.155 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111975.log.gz 2024-09-16T13:50:41.161 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136559.log.gz 2024-09-16T13:50:41.163 INFO:teuthology.orchestra.run.smithi172.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-client.admin.120109.log.gz 2024-09-16T13:50:41.164 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125948.log.gz 2024-09-16T13:50:41.168 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.129669.log.gz 2024-09-16T13:50:41.169 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125025.log.gz 2024-09-16T13:50:41.175 INFO:teuthology.orchestra.run.smithi172.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.97228.log.gz 2024-09-16T13:50:41.179 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104178.log.gz 2024-09-16T13:50:41.189 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128400.log.gz 2024-09-16T13:50:41.194 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92918.log.gz 2024-09-16T13:50:41.196 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113929.log.gz 2024-09-16T13:50:41.294 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.128993.log.gz 2024-09-16T13:50:41.297 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136113.log.gz 2024-09-16T13:50:41.303 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81558.log.gz 2024-09-16T13:50:41.306 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88328.log.gz 2024-09-16T13:50:41.308 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89761.log.gz 2024-09-16T13:50:41.310 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65397.log.gz 2024-09-16T13:50:41.311 INFO:teuthology.orchestra.run.smithi172.stderr: 79.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63677.log.gz 2024-09-16T13:50:41.311 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.118957.log.gz 2024-09-16T13:50:41.312 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.52529.log.gz 2024-09-16T13:50:41.315 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60770.log.gz 2024-09-16T13:50:41.318 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30724.log.gz 2024-09-16T13:50:41.326 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120383.log.gz 2024-09-16T13:50:41.329 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.84287.log.gz 2024-09-16T13:50:41.333 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.99504.log.gz 2024-09-16T13:50:41.338 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139224.log.gz 2024-09-16T13:50:41.342 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61810.log.gz 2024-09-16T13:50:41.358 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.94620.log.gz 2024-09-16T13:50:41.361 INFO:teuthology.orchestra.run.smithi172.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58857.log.gz 2024-09-16T13:50:41.366 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116793.log.gz 2024-09-16T13:50:41.374 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127319.log.gz 2024-09-16T13:50:41.377 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127485.log.gz 2024-09-16T13:50:41.380 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.116515.log.gz 2024-09-16T13:50:41.383 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.admin.26731.log.gz 2024-09-16T13:50:41.398 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93928.log.gz 2024-09-16T13:50:41.402 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.28848.log.gz 2024-09-16T13:50:41.403 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.49128.log.gz 2024-09-16T13:50:41.404 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.28932.log.gz 2024-09-16T13:50:41.441 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.59984.log.gz 2024-09-16T13:50:41.516 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.99299.log.gz 2024-09-16T13:50:41.531 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42854.log.gz 2024-09-16T13:50:41.532 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61831.log.gz 2024-09-16T13:50:41.533 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139165.log.gz 2024-09-16T13:50:41.534 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123221.log.gz 2024-09-16T13:50:41.535 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40224.log.gz 2024-09-16T13:50:41.538 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40560.log.gz 2024-09-16T13:50:41.539 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53880.log.gz 2024-09-16T13:50:41.544 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.87024.log.gz 2024-09-16T13:50:41.547 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.87096.log.gz 2024-09-16T13:50:41.549 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.87152.log.gz 2024-09-16T13:50:41.553 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.99842.log.gz 2024-09-16T13:50:41.554 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80931.log.gz 2024-09-16T13:50:41.556 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.84030.log.gz 2024-09-16T13:50:41.558 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.82221.log.gz 2024-09-16T13:50:41.565 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75523.log.gz 2024-09-16T13:50:41.568 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29505.log.gz 2024-09-16T13:50:41.575 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124266.log.gz 2024-09-16T13:50:41.578 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135338.log.gz 2024-09-16T13:50:41.579 INFO:teuthology.orchestra.run.smithi172.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.142499.log.gz 2024-09-16T13:50:41.582 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134578.log.gz 2024-09-16T13:50:41.586 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46540.log.gz 2024-09-16T13:50:41.588 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56335.log.gz 2024-09-16T13:50:41.608 INFO:teuthology.orchestra.run.smithi172.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.132103.log.gz 2024-09-16T13:50:41.608 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.132009.log.gz 2024-09-16T13:50:41.612 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104806.log.gz 2024-09-16T13:50:41.616 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52642.log.gz 2024-09-16T13:50:41.616 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52641.log.gz 2024-09-16T13:50:41.617 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.49558.log.gz 2024-09-16T13:50:41.621 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.38636.log.gz 2024-09-16T13:50:41.623 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57934.log.gz 2024-09-16T13:50:41.624 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56400.log.gz 2024-09-16T13:50:41.624 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62380.log.gz 2024-09-16T13:50:41.705 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89551.log.gz 2024-09-16T13:50:41.707 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77941.log.gz 2024-09-16T13:50:41.710 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102096.log.gz 2024-09-16T13:50:41.713 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77040.log.gz 2024-09-16T13:50:41.714 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75245.log.gz 2024-09-16T13:50:41.716 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103902.log.gz 2024-09-16T13:50:41.719 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.94283.log.gz 2024-09-16T13:50:41.722 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111303.log.gz 2024-09-16T13:50:41.724 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.135187.log.gz 2024-09-16T13:50:41.725 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.139500.log.gz 2024-09-16T13:50:41.727 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113047.log.gz 2024-09-16T13:50:41.728 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.130895.log.gz 2024-09-16T13:50:41.731 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.114663.log.gz 2024-09-16T13:50:41.733 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112207.log.gz 2024-09-16T13:50:41.735 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136431.log.gz 2024-09-16T13:50:41.736 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139081.log.gz 2024-09-16T13:50:41.737 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.118635.log.gz 2024-09-16T13:50:41.740 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.120711.log.gz 2024-09-16T13:50:41.744 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74496.log.gz 2024-09-16T13:50:41.746 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.99034.log.gz 2024-09-16T13:50:41.878 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29550.log.gz 2024-09-16T13:50:41.883 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31402.log.gz 2024-09-16T13:50:41.886 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54864.log.gz 2024-09-16T13:50:41.887 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68679.log.gz 2024-09-16T13:50:41.890 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45487.log.gz 2024-09-16T13:50:41.893 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45700.log.gz 2024-09-16T13:50:41.895 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.71272.log.gz 2024-09-16T13:50:41.895 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69813.log.gz 2024-09-16T13:50:41.897 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.109917.log.gz 2024-09-16T13:50:41.898 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58671.log.gz 2024-09-16T13:50:41.900 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52746.log.gz 2024-09-16T13:50:41.903 INFO:teuthology.orchestra.run.smithi172.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.98457.log.gz 2024-09-16T13:50:41.906 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115970.log.gz 2024-09-16T13:50:41.908 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126706.log.gz 2024-09-16T13:50:41.911 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103483.log.gz 2024-09-16T13:50:41.914 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.141605.log.gz 2024-09-16T13:50:41.918 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.70332.log.gz 2024-09-16T13:50:41.920 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64893.log.gz 2024-09-16T13:50:41.923 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67626.log.gz 2024-09-16T13:50:41.927 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88660.log.gz 2024-09-16T13:50:41.928 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.106711.log.gz 2024-09-16T13:50:41.930 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.106868.log.gz 2024-09-16T13:50:41.933 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100787.log.gz 2024-09-16T13:50:41.936 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.94941.log.gz 2024-09-16T13:50:41.941 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65184.log.gz 2024-09-16T13:50:41.943 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90558.log.gz 2024-09-16T13:50:41.947 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46687.log.gz 2024-09-16T13:50:41.950 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56545.log.gz 2024-09-16T13:50:41.952 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50387.log.gz 2024-09-16T13:50:41.954 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54424.log.gz 2024-09-16T13:50:41.958 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.68346.log.gz 2024-09-16T13:50:41.962 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135969.log.gz 2024-09-16T13:50:41.963 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.130304.log.gz 2024-09-16T13:50:41.963 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.130305.log.gz 2024-09-16T13:50:41.974 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.36289.log.gz 2024-09-16T13:50:41.979 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136642.log.gz 2024-09-16T13:50:41.979 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136599.log.gz 2024-09-16T13:50:41.979 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster1-client.admin.133556.log.gz 2024-09-16T13:50:41.983 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55958.log.gz 2024-09-16T13:50:41.990 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.28890.log.gz 2024-09-16T13:50:41.994 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.130727.log.gz 2024-09-16T13:50:42.009 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134153.log.gz 2024-09-16T13:50:42.010 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45486.log.gz 2024-09-16T13:50:42.013 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40630.log.gz 2024-09-16T13:50:42.014 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52155.log.gz 2024-09-16T13:50:42.017 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.28682.log.gz 2024-09-16T13:50:42.021 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45969.log.gz 2024-09-16T13:50:42.023 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55830.log.gz 2024-09-16T13:50:42.024 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55915.log.gz 2024-09-16T13:50:42.026 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46430.log.gz 2024-09-16T13:50:42.028 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster2-client.admin.73085.log.gz 2024-09-16T13:50:42.029 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77670.log.gz 2024-09-16T13:50:42.035 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102706.log.gz 2024-09-16T13:50:42.037 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42706.log.gz 2024-09-16T13:50:42.040 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.61623.log.gz 2024-09-16T13:50:42.044 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31610.log.gz 2024-09-16T13:50:42.046 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69396.log.gz 2024-09-16T13:50:42.050 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105206.log.gz 2024-09-16T13:50:42.052 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89340.log.gz 2024-09-16T13:50:42.054 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90140.log.gz 2024-09-16T13:50:42.055 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90222.log.gz 2024-09-16T13:50:42.057 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90325.log.gz 2024-09-16T13:50:42.058 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90351.log.gz 2024-09-16T13:50:42.060 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76778.log.gz 2024-09-16T13:50:42.061 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87844.log.gz 2024-09-16T13:50:42.062 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.120477.log.gz 2024-09-16T13:50:42.064 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103016.log.gz 2024-09-16T13:50:42.066 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91019.log.gz 2024-09-16T13:50:42.069 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104153.log.gz 2024-09-16T13:50:42.072 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136197.log.gz 2024-09-16T13:50:42.075 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136979.log.gz 2024-09-16T13:50:42.077 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138323.log.gz 2024-09-16T13:50:42.078 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137229.log.gz 2024-09-16T13:50:42.079 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137964.log.gz 2024-09-16T13:50:42.081 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127426.log.gz 2024-09-16T13:50:42.083 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112730.log.gz 2024-09-16T13:50:42.084 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125527.log.gz 2024-09-16T13:50:42.086 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127507.log.gz 2024-09-16T13:50:42.089 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125865.log.gz 2024-09-16T13:50:42.091 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.133915.log.gz 2024-09-16T13:50:42.093 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124354.log.gz 2024-09-16T13:50:42.095 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.130077.log.gz 2024-09-16T13:50:42.099 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111499.log.gz 2024-09-16T13:50:42.112 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.121876.log.gz 2024-09-16T13:50:42.204 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74915.log.gz 2024-09-16T13:50:42.207 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42853.log.gz 2024-09-16T13:50:42.210 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.61878.log.gz 2024-09-16T13:50:42.220 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80194.log.gz 2024-09-16T13:50:42.221 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.85073.log.gz 2024-09-16T13:50:42.222 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.84783.log.gz 2024-09-16T13:50:42.226 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78696.log.gz 2024-09-16T13:50:42.227 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79454.log.gz 2024-09-16T13:50:42.227 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.83836.log.gz 2024-09-16T13:50:42.231 INFO:teuthology.orchestra.run.smithi172.stderr: 90.2% -- replaced with /var/log/ceph/cluster1-client.mirror.105629.log.gz 2024-09-16T13:50:42.231 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104388.log.gz 2024-09-16T13:50:42.234 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45759.log.gz 2024-09-16T13:50:42.236 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65522.log.gz 2024-09-16T13:50:42.239 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44073.log.gz 2024-09-16T13:50:42.239 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44176.log.gz 2024-09-16T13:50:42.241 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127423.log.gz 2024-09-16T13:50:42.245 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.33159.log.gz 2024-09-16T13:50:42.251 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118849.log.gz 2024-09-16T13:50:42.252 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125042.log.gz 2024-09-16T13:50:42.256 INFO:teuthology.orchestra.run.smithi172.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.119881.log.gz 2024-09-16T13:50:42.257 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33411.log.gz 2024-09-16T13:50:42.259 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.96275.log.gz 2024-09-16T13:50:42.262 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90745.log.gz 2024-09-16T13:50:42.263 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.84380.log.gz 2024-09-16T13:50:42.267 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.110756.log.gz 2024-09-16T13:50:42.270 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52153.log.gz 2024-09-16T13:50:42.297 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122040.log.gz 2024-09-16T13:50:42.375 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57116.log.gz 2024-09-16T13:50:42.377 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56040.log.gz 2024-09-16T13:50:42.378 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45910.log.gz 2024-09-16T13:50:42.380 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51904.log.gz 2024-09-16T13:50:42.381 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.107385.log.gz 2024-09-16T13:50:42.383 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104489.log.gz 2024-09-16T13:50:42.386 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31612.log.gz 2024-09-16T13:50:42.388 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31736.log.gz 2024-09-16T13:50:42.395 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58124.log.gz 2024-09-16T13:50:42.397 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66318.log.gz 2024-09-16T13:50:42.401 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77436.log.gz 2024-09-16T13:50:42.402 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77521.log.gz 2024-09-16T13:50:42.404 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88881.log.gz 2024-09-16T13:50:42.408 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.109086.log.gz 2024-09-16T13:50:42.409 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89845.log.gz 2024-09-16T13:50:42.412 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122753.log.gz 2024-09-16T13:50:42.414 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116581.log.gz 2024-09-16T13:50:42.415 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115403.log.gz 2024-09-16T13:50:42.426 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140086.log.gz 2024-09-16T13:50:42.431 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103102.log.gz 2024-09-16T13:50:42.435 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64733.log.gz 2024-09-16T13:50:42.437 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53752.log.gz 2024-09-16T13:50:42.439 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.48115.log.gz 2024-09-16T13:50:42.452 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.82332.log.gz 2024-09-16T13:50:42.464 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122218.log.gz 2024-09-16T13:50:42.515 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102493.log.gz 2024-09-16T13:50:42.548 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.39138.log.gz 2024-09-16T13:50:42.585 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77461.log.gz 2024-09-16T13:50:42.589 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31192.log.gz 2024-09-16T13:50:42.589 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.40386.log.gz 2024-09-16T13:50:42.592 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.98834.log.gz 2024-09-16T13:50:42.598 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55745.log.gz 2024-09-16T13:50:42.599 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68318.log.gz 2024-09-16T13:50:42.602 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.98323.log.gz 2024-09-16T13:50:42.603 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.117805.log.gz 2024-09-16T13:50:42.605 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116558.log.gz 2024-09-16T13:50:42.614 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124034.log.gz 2024-09-16T13:50:42.617 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78044.log.gz 2024-09-16T13:50:42.619 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.132164.log.gz 2024-09-16T13:50:42.619 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.132217.log.gz 2024-09-16T13:50:42.637 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139121.log.gz 2024-09-16T13:50:42.638 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.73491.log.gz 2024-09-16T13:50:42.641 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101379.log.gz 2024-09-16T13:50:42.645 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38796.log.gz 2024-09-16T13:50:42.648 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65688.log.gz 2024-09-16T13:50:42.652 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.33080.log.gz 2024-09-16T13:50:42.657 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31941.log.gz 2024-09-16T13:50:42.660 INFO:teuthology.orchestra.run.smithi172.stderr: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62059.log.gz 2024-09-16T13:50:42.664 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.71676.log.gz 2024-09-16T13:50:42.722 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.108129.log.gz 2024-09-16T13:50:42.724 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45699.log.gz 2024-09-16T13:50:42.726 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43173.log.gz 2024-09-16T13:50:42.729 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67414.log.gz 2024-09-16T13:50:42.731 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69475.log.gz 2024-09-16T13:50:42.732 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.71675.log.gz 2024-09-16T13:50:42.733 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67158.log.gz 2024-09-16T13:50:42.736 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34044.log.gz 2024-09-16T13:50:42.738 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44743.log.gz 2024-09-16T13:50:42.741 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32784.log.gz 2024-09-16T13:50:42.743 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41866.log.gz 2024-09-16T13:50:42.747 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101171.log.gz 2024-09-16T13:50:42.748 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.119425.log.gz 2024-09-16T13:50:42.749 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.119539.log.gz 2024-09-16T13:50:42.757 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93293.log.gz 2024-09-16T13:50:42.760 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137122.log.gz 2024-09-16T13:50:42.762 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113193.log.gz 2024-09-16T13:50:42.763 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124605.log.gz 2024-09-16T13:50:42.766 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122338.log.gz 2024-09-16T13:50:42.768 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122604.log.gz 2024-09-16T13:50:42.769 INFO:teuthology.orchestra.run.smithi172.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.131989.log.gz 2024-09-16T13:50:42.778 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.98723.log.gz 2024-09-16T13:50:42.780 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.99159.log.gz 2024-09-16T13:50:42.784 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134710.log.gz 2024-09-16T13:50:42.787 INFO:teuthology.orchestra.run.smithi172.stderr: 19.6% -- replaced with /var/log/ceph/cluster1-client.admin.133379.log.gz 2024-09-16T13:50:42.798 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90851.log.gz 2024-09-16T13:50:42.891 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46284.log.gz 2024-09-16T13:50:42.894 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45801.log.gz 2024-09-16T13:50:42.895 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56189.log.gz 2024-09-16T13:50:42.900 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40927.log.gz 2024-09-16T13:50:42.901 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69055.log.gz 2024-09-16T13:50:42.907 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43545.log.gz 2024-09-16T13:50:42.909 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104619.log.gz 2024-09-16T13:50:42.911 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.83796.log.gz 2024-09-16T13:50:42.918 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.97088.log.gz 2024-09-16T13:50:42.919 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89381.log.gz 2024-09-16T13:50:42.922 INFO:teuthology.orchestra.run.smithi172.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-client.admin.50548.log.gz 2024-09-16T13:50:42.923 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.115781.log.gz 2024-09-16T13:50:42.926 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78824.log.gz 2024-09-16T13:50:42.929 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.142548.log.gz 2024-09-16T13:50:42.931 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113361.log.gz 2024-09-16T13:50:42.931 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113447.log.gz 2024-09-16T13:50:42.932 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123975.log.gz 2024-09-16T13:50:42.935 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122624.log.gz 2024-09-16T13:50:42.937 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123303.log.gz 2024-09-16T13:50:42.941 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102009.log.gz 2024-09-16T13:50:42.943 INFO:teuthology.orchestra.run.smithi172.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.109025.log.gz 2024-09-16T13:50:42.947 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.127716.log.gz 2024-09-16T13:50:42.951 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.85408.log.gz 2024-09-16T13:50:42.955 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104365.log.gz 2024-09-16T13:50:42.961 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100328.log.gz 2024-09-16T13:50:42.965 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89636.log.gz 2024-09-16T13:50:42.972 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29185.log.gz 2024-09-16T13:50:42.977 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32452.log.gz 2024-09-16T13:50:42.982 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44855.log.gz 2024-09-16T13:50:42.985 INFO:teuthology.orchestra.run.smithi172.stderr: 56.6% -- replaced with /var/log/ceph/cluster1-client.mirror.49931.log.gz 2024-09-16T13:50:43.048 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63277.log.gz 2024-09-16T13:50:43.049 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.83620.log.gz 2024-09-16T13:50:43.049 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78781.log.gz 2024-09-16T13:50:43.052 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31611.log.gz 2024-09-16T13:50:43.053 INFO:teuthology.orchestra.run.smithi172.stderr: 19.2% -- replaced with /var/log/ceph/cluster1-client.admin.27814.log.gz 2024-09-16T13:50:43.058 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42286.log.gz 2024-09-16T13:50:43.070 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102306.log.gz 2024-09-16T13:50:43.074 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88465.log.gz 2024-09-16T13:50:43.075 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.86374.log.gz 2024-09-16T13:50:43.078 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101589.log.gz 2024-09-16T13:50:43.080 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90141.log.gz 2024-09-16T13:50:43.082 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77981.log.gz 2024-09-16T13:50:43.084 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87667.log.gz 2024-09-16T13:50:43.085 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127425.log.gz 2024-09-16T13:50:43.089 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99986.log.gz 2024-09-16T13:50:43.091 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65844.log.gz 2024-09-16T13:50:43.094 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42642.log.gz 2024-09-16T13:50:43.097 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42852.log.gz 2024-09-16T13:50:43.102 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.115021.log.gz 2024-09-16T13:50:43.104 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128292.log.gz 2024-09-16T13:50:43.107 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126582.log.gz 2024-09-16T13:50:43.111 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.130802.log.gz 2024-09-16T13:50:43.112 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120290.log.gz 2024-09-16T13:50:43.115 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster1-client.admin.132477.log.gz 2024-09-16T13:50:43.223 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99982.log.gz 2024-09-16T13:50:43.247 INFO:teuthology.orchestra.run.smithi172.stderr: 17.9% -- replaced with /var/log/ceph/cluster2-client.admin.118954.log.gz 2024-09-16T13:50:43.247 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.118959.log.gz 2024-09-16T13:50:43.248 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.97020.log.gz 2024-09-16T13:50:43.263 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.95781.log.gz 2024-09-16T13:50:43.266 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.72775.log.gz 2024-09-16T13:50:43.266 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.75764.log.gz 2024-09-16T13:50:43.269 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.77522.log.gz 2024-09-16T13:50:43.276 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69687.log.gz 2024-09-16T13:50:43.277 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.37059.log.gz 2024-09-16T13:50:43.278 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.61965.log.gz 2024-09-16T13:50:43.282 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.36481.log.gz 2024-09-16T13:50:43.282 INFO:teuthology.orchestra.run.smithi172.stderr: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.39690.log.gz 2024-09-16T13:50:43.285 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.36482.log.gz 2024-09-16T13:50:43.287 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43335.log.gz 2024-09-16T13:50:43.289 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46121.log.gz 2024-09-16T13:50:43.291 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.31479.log.gz 2024-09-16T13:50:43.294 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35163.log.gz 2024-09-16T13:50:43.297 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.52106.log.gz 2024-09-16T13:50:43.300 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33583.log.gz 2024-09-16T13:50:43.304 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136011.log.gz 2024-09-16T13:50:43.307 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91608.log.gz 2024-09-16T13:50:43.319 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster2-client.admin.49140.log.gz 2024-09-16T13:50:43.323 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.108288.log.gz 2024-09-16T13:50:43.327 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138424.log.gz 2024-09-16T13:50:43.330 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112671.log.gz 2024-09-16T13:50:43.338 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40787.log.gz 2024-09-16T13:50:43.340 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53331.log.gz 2024-09-16T13:50:43.343 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52899.log.gz 2024-09-16T13:50:43.346 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31144.log.gz 2024-09-16T13:50:43.349 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55853.log.gz 2024-09-16T13:50:43.352 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.58791.log.gz 2024-09-16T13:50:43.353 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.61764.log.gz 2024-09-16T13:50:43.358 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.60513.log.gz 2024-09-16T13:50:43.360 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63737.log.gz 2024-09-16T13:50:43.360 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63741.log.gz 2024-09-16T13:50:43.363 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52217.log.gz 2024-09-16T13:50:43.366 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70110.log.gz 2024-09-16T13:50:43.379 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76327.log.gz 2024-09-16T13:50:43.381 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78510.log.gz 2024-09-16T13:50:43.386 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.84569.log.gz 2024-09-16T13:50:43.388 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.72165.log.gz 2024-09-16T13:50:43.392 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87579.log.gz 2024-09-16T13:50:43.397 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89004.log.gz 2024-09-16T13:50:43.402 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105332.log.gz 2024-09-16T13:50:43.405 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.73511.log.gz 2024-09-16T13:50:43.406 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.115716.log.gz 2024-09-16T13:50:43.490 INFO:teuthology.orchestra.run.smithi172.stderr: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.119720.log.gz 2024-09-16T13:50:43.491 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.122220.log.gz 2024-09-16T13:50:43.493 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126163.log.gz 2024-09-16T13:50:43.497 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135063.log.gz 2024-09-16T13:50:43.502 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136491.log.gz 2024-09-16T13:50:43.503 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136558.log.gz 2024-09-16T13:50:43.505 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112000.log.gz 2024-09-16T13:50:43.510 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.142324.log.gz 2024-09-16T13:50:43.517 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.117645.log.gz 2024-09-16T13:50:43.522 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57642.log.gz 2024-09-16T13:50:43.523 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57665.log.gz 2024-09-16T13:50:43.530 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.27865.log.gz 2024-09-16T13:50:43.531 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.120917.log.gz 2024-09-16T13:50:43.531 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.29842.log.gz 2024-09-16T13:50:43.568 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42754.log.gz 2024-09-16T13:50:43.598 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.85573.log.gz 2024-09-16T13:50:43.610 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.74596.log.gz 2024-09-16T13:50:43.613 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92030.log.gz 2024-09-16T13:50:43.614 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81098.log.gz 2024-09-16T13:50:43.624 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.96974.log.gz 2024-09-16T13:50:43.625 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32110.log.gz 2024-09-16T13:50:43.629 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45278.log.gz 2024-09-16T13:50:43.631 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42222.log.gz 2024-09-16T13:50:43.634 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32239.log.gz 2024-09-16T13:50:43.635 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.129764.log.gz 2024-09-16T13:50:43.637 INFO:teuthology.orchestra.run.smithi172.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.admin.71898.log.gz 2024-09-16T13:50:43.643 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79455.log.gz 2024-09-16T13:50:43.646 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126453.log.gz 2024-09-16T13:50:43.649 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116074.log.gz 2024-09-16T13:50:43.651 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.116288.log.gz 2024-09-16T13:50:43.655 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.133513.log.gz 2024-09-16T13:50:43.659 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30934.log.gz 2024-09-16T13:50:43.660 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30978.log.gz 2024-09-16T13:50:43.664 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101232.log.gz 2024-09-16T13:50:43.667 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89297.log.gz 2024-09-16T13:50:43.670 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.73880.log.gz 2024-09-16T13:50:43.674 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90348.log.gz 2024-09-16T13:50:43.677 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58019.log.gz 2024-09-16T13:50:43.756 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42602.log.gz 2024-09-16T13:50:43.758 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51357.log.gz 2024-09-16T13:50:43.762 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.28975.log.gz 2024-09-16T13:50:43.767 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.50179.log.gz 2024-09-16T13:50:43.770 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53669.log.gz 2024-09-16T13:50:43.773 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53249.log.gz 2024-09-16T13:50:43.774 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53396.log.gz 2024-09-16T13:50:43.779 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.27970.log.gz 2024-09-16T13:50:43.780 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.28084.log.gz 2024-09-16T13:50:43.791 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29716.log.gz 2024-09-16T13:50:43.802 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46431.log.gz 2024-09-16T13:50:43.804 INFO:teuthology.orchestra.run.smithi172.stderr: 17.4% -- replaced with /var/log/ceph/cluster1-client.admin.75197.log.gz 2024-09-16T13:50:43.806 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79871.log.gz 2024-09-16T13:50:43.809 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92410.log.gz 2024-09-16T13:50:43.812 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102916.log.gz 2024-09-16T13:50:43.815 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116687.log.gz 2024-09-16T13:50:43.818 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.95736.log.gz 2024-09-16T13:50:43.820 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.108402.log.gz 2024-09-16T13:50:43.820 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126642.log.gz 2024-09-16T13:50:43.824 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66444.log.gz 2024-09-16T13:50:43.836 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.122492.log.gz 2024-09-16T13:50:43.837 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.135318.log.gz 2024-09-16T13:50:43.837 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136072.log.gz 2024-09-16T13:50:43.841 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117033.log.gz 2024-09-16T13:50:43.842 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.132860.log.gz 2024-09-16T13:50:43.843 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.133046.log.gz 2024-09-16T13:50:43.845 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.111215.log.gz 2024-09-16T13:50:43.861 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93588.log.gz 2024-09-16T13:50:43.861 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93672.log.gz 2024-09-16T13:50:43.863 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93797.log.gz 2024-09-16T13:50:43.869 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64848.log.gz 2024-09-16T13:50:43.967 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55325.log.gz 2024-09-16T13:50:43.969 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55432.log.gz 2024-09-16T13:50:43.971 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80989.log.gz 2024-09-16T13:50:43.972 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.118311.log.gz 2024-09-16T13:50:43.973 INFO:teuthology.orchestra.run.smithi172.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.admin.84949.log.gz 2024-09-16T13:50:43.976 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.118955.log.gz 2024-09-16T13:50:43.978 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51857.log.gz 2024-09-16T13:50:43.989 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33290.log.gz 2024-09-16T13:50:43.993 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40069.log.gz 2024-09-16T13:50:43.996 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93126.log.gz 2024-09-16T13:50:43.999 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80588.log.gz 2024-09-16T13:50:44.003 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.128139.log.gz 2024-09-16T13:50:44.003 INFO:teuthology.orchestra.run.smithi172.stderr: 17.9% -- replaced with /var/log/ceph/cluster2-client.admin.142554.log.gz 2024-09-16T13:50:44.008 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77982.log.gz 2024-09-16T13:50:44.011 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137187.log.gz 2024-09-16T13:50:44.012 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.130422.log.gz 2024-09-16T13:50:44.016 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.121448.log.gz 2024-09-16T13:50:44.020 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118488.log.gz 2024-09-16T13:50:44.022 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.48119.log.gz 2024-09-16T13:50:44.029 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50843.log.gz 2024-09-16T13:50:44.034 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.118841.log.gz 2024-09-16T13:50:44.067 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42432.log.gz 2024-09-16T13:50:44.075 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54091.log.gz 2024-09-16T13:50:44.078 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54302.log.gz 2024-09-16T13:50:44.080 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34343.log.gz 2024-09-16T13:50:44.083 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58463.log.gz 2024-09-16T13:50:44.092 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43755.log.gz 2024-09-16T13:50:44.097 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78486.log.gz 2024-09-16T13:50:44.097 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61924.log.gz 2024-09-16T13:50:44.097 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88962.log.gz 2024-09-16T13:50:44.101 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50594.log.gz 2024-09-16T13:50:44.104 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68106.log.gz 2024-09-16T13:50:44.108 INFO:teuthology.orchestra.run.smithi172.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63347.log.gz 2024-09-16T13:50:44.114 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91481.log.gz 2024-09-16T13:50:44.118 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124395.log.gz 2024-09-16T13:50:44.120 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111156.log.gz 2024-09-16T13:50:44.126 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80088.log.gz 2024-09-16T13:50:44.135 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91232.log.gz 2024-09-16T13:50:44.139 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114707.log.gz 2024-09-16T13:50:44.144 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.115043.log.gz 2024-09-16T13:50:44.226 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64736.log.gz 2024-09-16T13:50:44.235 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.121001.log.gz 2024-09-16T13:50:44.243 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67788.log.gz 2024-09-16T13:50:44.244 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41914.log.gz 2024-09-16T13:50:44.246 INFO:teuthology.orchestra.run.smithi172.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49890.log.gz 2024-09-16T13:50:44.249 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51220.log.gz 2024-09-16T13:50:44.250 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60945.log.gz 2024-09-16T13:50:44.262 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.108630.log.gz 2024-09-16T13:50:44.265 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92158.log.gz 2024-09-16T13:50:44.266 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.106571.log.gz 2024-09-16T13:50:44.268 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78782.log.gz 2024-09-16T13:50:44.270 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104786.log.gz 2024-09-16T13:50:44.277 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135927.log.gz 2024-09-16T13:50:44.295 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111833.log.gz 2024-09-16T13:50:44.299 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.37067.log.gz 2024-09-16T13:50:44.300 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.59985.log.gz 2024-09-16T13:50:44.432 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29186.log.gz 2024-09-16T13:50:44.433 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.36288.log.gz 2024-09-16T13:50:44.437 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100391.log.gz 2024-09-16T13:50:44.439 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68255.log.gz 2024-09-16T13:50:44.441 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131211.log.gz 2024-09-16T13:50:44.442 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33512.log.gz 2024-09-16T13:50:44.445 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137901.log.gz 2024-09-16T13:50:44.448 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113977.log.gz 2024-09-16T13:50:44.450 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.118333.log.gz 2024-09-16T13:50:44.452 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.133218.log.gz 2024-09-16T13:50:44.456 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38910.log.gz 2024-09-16T13:50:44.460 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62266.log.gz 2024-09-16T13:50:44.463 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115193.log.gz 2024-09-16T13:50:44.465 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93336.log.gz 2024-09-16T13:50:44.466 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.107540.log.gz 2024-09-16T13:50:44.488 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78612.log.gz 2024-09-16T13:50:44.496 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100032.log.gz 2024-09-16T13:50:44.499 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54655.log.gz 2024-09-16T13:50:44.500 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62588.log.gz 2024-09-16T13:50:44.505 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100599.log.gz 2024-09-16T13:50:44.510 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.121313.log.gz 2024-09-16T13:50:44.514 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116645.log.gz 2024-09-16T13:50:44.517 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135760.log.gz 2024-09-16T13:50:44.520 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123873.log.gz 2024-09-16T13:50:44.526 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63717.log.gz 2024-09-16T13:50:44.527 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63783.log.gz 2024-09-16T13:50:44.532 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112143.log.gz 2024-09-16T13:50:44.534 INFO:teuthology.orchestra.run.smithi172.stderr: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76031.log.gz 2024-09-16T13:50:44.545 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64953.log.gz 2024-09-16T13:50:44.548 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68846.log.gz 2024-09-16T13:50:44.549 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.36479.log.gz 2024-09-16T13:50:44.549 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.106545.log.gz 2024-09-16T13:50:44.555 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69398.log.gz 2024-09-16T13:50:44.558 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40323.log.gz 2024-09-16T13:50:44.558 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.36606.log.gz 2024-09-16T13:50:44.561 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.107868.log.gz 2024-09-16T13:50:44.565 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster1-client.admin.110076.log.gz 2024-09-16T13:50:44.568 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.98421.log.gz 2024-09-16T13:50:44.568 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98265.log.gz 2024-09-16T13:50:44.569 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98483.log.gz 2024-09-16T13:50:44.707 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60765.log.gz 2024-09-16T13:50:44.709 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45019.log.gz 2024-09-16T13:50:44.712 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35035.log.gz 2024-09-16T13:50:44.713 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34254.log.gz 2024-09-16T13:50:44.716 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39045.log.gz 2024-09-16T13:50:44.721 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92621.log.gz 2024-09-16T13:50:44.723 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88326.log.gz 2024-09-16T13:50:44.728 INFO:teuthology.orchestra.run.smithi172.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.97202.log.gz 2024-09-16T13:50:44.740 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139332.log.gz 2024-09-16T13:50:49.011 INFO:teuthology.orchestra.run.smithi018.stderr: 90.6% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-09-16T13:50:49.877 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66214.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.41128.log.gz 2024-09-16T13:50:49.877 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:49.877 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.59554.log.gz 2024-09-16T13:50:49.877 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75429.log.gz 2024-09-16T13:50:49.877 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34465.log.gz 2024-09-16T13:50:49.877 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49091.log.gz 2024-09-16T13:50:49.878 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.133709.log.gz 2024-09-16T13:50:49.878 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79982.log.gz 2024-09-16T13:50:49.878 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88921.log.gz 2024-09-16T13:50:49.878 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52816.log.gz 2024-09-16T13:50:49.878 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.110533.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.54088.log.gz 2024-09-16T13:50:49.878 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:50:49.878 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.85690.log.gz 2024-09-16T13:50:49.878 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105416.log.gz 2024-09-16T13:50:49.879 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92789.log.gz 2024-09-16T13:50:49.885 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.36222.log.gz 2024-09-16T13:50:49.885 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57876.log.gz 2024-09-16T13:50:49.885 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32032.log.gz 2024-09-16T13:50:49.885 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78404.log.gz 2024-09-16T13:50:49.885 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.130212.log.gz 2024-09-16T13:51:01.162 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29841.log.gz 2024-09-16T13:51:01.163 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.110446.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.59774.log.gz 2024-09-16T13:51:01.163 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.163 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.82271.log.gz 2024-09-16T13:51:01.163 INFO:teuthology.orchestra.run.smithi172.stderr: 17.9% -- replaced with /var/log/ceph/cluster1-client.admin.110604.log.gz 2024-09-16T13:51:01.163 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.129799.log.gz 2024-09-16T13:51:01.163 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.27926.log.gz 2024-09-16T13:51:01.163 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.98527.log.gz 2024-09-16T13:51:01.163 INFO:teuthology.orchestra.run.smithi172.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.63503.log.gz 2024-09-16T13:51:01.163 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104678.log.gz 2024-09-16T13:51:01.163 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43546.log.gz 2024-09-16T13:51:01.163 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43903.log.gz 2024-09-16T13:51:01.164 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.130431.log.gz 2024-09-16T13:51:01.164 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.28019.log.gz 2024-09-16T13:51:01.164 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104177.log.gz 2024-09-16T13:51:01.164 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109045.log.gz 2024-09-16T13:51:01.164 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.109066.log.gz 2024-09-16T13:51:01.164 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.62193.log.gz 2024-09-16T13:51:01.164 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40833.log.gz 2024-09-16T13:51:01.164 INFO:teuthology.orchestra.run.smithi172.stderr: 80.1% -- replaced with /var/log/ceph/cluster1-client.mirror.110652.log.gz 2024-09-16T13:51:01.164 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88702.log.gz 2024-09-16T13:51:01.164 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% 57.7% 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.129536.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.26850.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.93421.log.gz 2024-09-16T13:51:01.164 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.164 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.172 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69941.log.gz 2024-09-16T13:51:01.172 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102385.log.gz 2024-09-16T13:51:01.173 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104533.log.gz 2024-09-16T13:51:01.173 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115359.log.gz 2024-09-16T13:51:01.173 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32363.log.gz 2024-09-16T13:51:01.173 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50045.log.gz 2024-09-16T13:51:01.173 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125443.log.gz 2024-09-16T13:51:01.173 INFO:teuthology.orchestra.run.smithi172.stderr: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.131916.log.gz 2024-09-16T13:51:01.173 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45651.log.gz 2024-09-16T13:51:01.173 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.73078.log.gz 90.0% 2024-09-16T13:51:01.173 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.32660.log.gz 2024-09-16T13:51:01.173 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster2-client.admin.72831.log.gz 2024-09-16T13:51:01.173 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.117983.log.gz 2024-09-16T13:51:01.173 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.39092.log.gz 2024-09-16T13:51:01.173 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.142552.log.gz 2024-09-16T13:51:01.174 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32571.log.gz 2024-09-16T13:51:01.174 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115884.log.gz 2024-09-16T13:51:01.174 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58684.log.gz 87.4% 2024-09-16T13:51:01.174 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.32659.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.95115.log.gz 2024-09-16T13:51:01.174 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.174 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29226.log.gz 2024-09-16T13:51:01.174 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78826.log.gz 2024-09-16T13:51:01.174 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.87509.log.gz 2024-09-16T13:51:01.174 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.132314.log.gz 2024-09-16T13:51:01.174 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126242.log.gz 2024-09-16T13:51:01.174 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64397.log.gz 2024-09-16T13:51:01.174 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30178.log.gz 2024-09-16T13:51:01.174 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.50863.log.gz 2024-09-16T13:51:01.175 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29101.log.gz 2024-09-16T13:51:01.175 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55851.log.gz 2024-09-16T13:51:01.175 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.38408.log.gz 2024-09-16T13:51:01.175 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79326.log.gz 2024-09-16T13:51:01.175 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67368.log.gz 2024-09-16T13:51:01.175 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81035.log.gz 2024-09-16T13:51:01.175 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.96792.log.gz 2024-09-16T13:51:01.175 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.48288.log.gz 2024-09-16T13:51:01.175 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.57218.log.gz 2024-09-16T13:51:01.175 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46010.log.gz 2024-09-16T13:51:01.175 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.95218.log.gz 2024-09-16T13:51:01.175 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65374.log.gz 2024-09-16T13:51:01.175 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136178.log.gz 2024-09-16T13:51:01.175 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.141602.log.gz 2024-09-16T13:51:01.175 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90642.log.gz 2024-09-16T13:51:01.176 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.73076.log.gz 2024-09-16T13:51:01.176 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.40496.log.gz 2024-09-16T13:51:01.176 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.48490.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.47895.log.gz 2024-09-16T13:51:01.176 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.176 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79724.log.gz 2024-09-16T13:51:01.176 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114937.log.gz 2024-09-16T13:51:01.176 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38703.log.gz 2024-09-16T13:51:01.176 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.102283.log.gz 2024-09-16T13:51:01.176 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69731.log.gz 2024-09-16T13:51:01.176 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.38065.log.gz 2024-09-16T13:51:01.176 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.75896.log.gz 2024-09-16T13:51:01.176 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.121839.log.gz 2024-09-16T13:51:01.176 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.138593.log.gz 2024-09-16T13:51:01.176 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102806.log.gz 2024-09-16T13:51:01.177 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85991.log.gz 2024-09-16T13:51:01.177 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.59767.log.gz 2024-09-16T13:51:01.177 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50708.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.71737.log.gz 2024-09-16T13:51:01.177 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.177 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.118721.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.142538.log.gz 2024-09-16T13:51:01.177 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.177 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.50521.log.gz 2024-09-16T13:51:01.177 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.95428.log.gz 2024-09-16T13:51:01.177 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112462.log.gz 2024-09-16T13:51:01.177 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73698.log.gz 2024-09-16T13:51:01.177 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101882.log.gz 2024-09-16T13:51:01.177 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57599.log.gz 2024-09-16T13:51:01.177 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.85831.log.gz 2024-09-16T13:51:01.178 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124773.log.gz 2024-09-16T13:51:01.178 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101674.log.gz 2024-09-16T13:51:01.178 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88534.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.121442.log.gz 2024-09-16T13:51:01.178 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.178 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.131781.log.gz 2024-09-16T13:51:01.178 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64627.log.gz 2024-09-16T13:51:01.178 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73584.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.104954.log.gz 2024-09-16T13:51:01.178 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.178 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90812.log.gz 2024-09-16T13:51:01.178 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.72328.log.gz 2024-09-16T13:51:01.178 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.47900.log.gz 2024-09-16T13:51:01.178 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.118949.log.gz 2024-09-16T13:51:01.178 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.109314.log.gz 2024-09-16T13:51:01.178 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.48238.log.gz 2024-09-16T13:51:01.178 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.120574.log.gz 2024-09-16T13:51:01.179 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.39642.log.gz 2024-09-16T13:51:01.179 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.48052.log.gz 2024-09-16T13:51:01.179 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38682.log.gz 2024-09-16T13:51:01.179 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79349.log.gz 2024-09-16T13:51:01.179 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139433.log.gz 2024-09-16T13:51:01.179 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81667.log.gz 89.6% 2024-09-16T13:51:01.179 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.132294.log.gz 2024-09-16T13:51:01.179 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32780.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.41129.log.gz 2024-09-16T13:51:01.180 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.180 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.109540.log.gz 2024-09-16T13:51:01.180 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115653.log.gz 2024-09-16T13:51:01.180 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125614.log.gz 2024-09-16T13:51:01.180 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69607.log.gz 2024-09-16T13:51:01.180 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.132705.log.gz 2024-09-16T13:51:01.180 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51393.log.gz 2024-09-16T13:51:01.180 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63432.log.gz 2024-09-16T13:51:01.180 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115505.log.gz 2024-09-16T13:51:01.180 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138533.log.gz 2024-09-16T13:51:01.180 INFO:teuthology.orchestra.run.smithi172.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38890.log.gz 2024-09-16T13:51:01.181 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104396.log.gz 2024-09-16T13:51:01.181 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78722.log.gz 2024-09-16T13:51:01.181 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101734.log.gz 2024-09-16T13:51:01.181 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.109180.log.gz 2024-09-16T13:51:01.181 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30470.log.gz 2024-09-16T13:51:01.181 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.37041.log.gz 2024-09-16T13:51:01.181 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.106686.log.gz 2024-09-16T13:51:01.181 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81917.log.gz 2024-09-16T13:51:01.181 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108931.log.gz 2024-09-16T13:51:01.181 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103339.log.gz 2024-09-16T13:51:01.181 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.72701.log.gz 2024-09-16T13:51:01.181 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% 87.3% 90.0% -- replaced with /var/log/ceph/cluster2-client.admin.85069.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.132787.log.gz 2024-09-16T13:51:01.181 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.182 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.138660.log.gz 2024-09-16T13:51:01.182 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114979.log.gz 2024-09-16T13:51:01.182 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138491.log.gz 2024-09-16T13:51:01.182 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66576.log.gz 2024-09-16T13:51:01.182 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102707.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.54299.log.gz 2024-09-16T13:51:01.182 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.182 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% 89.8% -- replaced with /var/log/ceph/cluster2-client.admin.71900.log.gz 2024-09-16T13:51:01.182 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.114872.log.gz 2024-09-16T13:51:01.182 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.129535.log.gz 2024-09-16T13:51:01.479 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30303.log.gz 2024-09-16T13:51:01.479 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.135231.log.gz 2024-09-16T13:51:01.479 INFO:teuthology.orchestra.run.smithi172.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50460.log.gz 2024-09-16T13:51:01.479 INFO:teuthology.orchestra.run.smithi172.stderr: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.132123.log.gz 2024-09-16T13:51:01.479 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.107867.log.gz 2024-09-16T13:51:01.479 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.96279.log.gz 17.4% 2024-09-16T13:51:01.479 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.96273.log.gz 2024-09-16T13:51:01.479 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89171.log.gz 2024-09-16T13:51:01.479 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111789.log.gz 2024-09-16T13:51:01.479 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109273.log.gz 2024-09-16T13:51:01.480 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.106318.log.gz 2024-09-16T13:51:01.480 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster2-client.admin.73068.log.gz 2024-09-16T13:51:01.480 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.117041.log.gz 2024-09-16T13:51:01.480 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113554.log.gz 2024-09-16T13:51:01.480 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136390.log.gz 2024-09-16T13:51:01.480 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108703.log.gz 2024-09-16T13:51:01.480 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50366.log.gz 2024-09-16T13:51:01.480 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.88847.log.gz 2024-09-16T13:51:01.481 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93713.log.gz 2024-09-16T13:51:01.481 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.105715.log.gz 2024-09-16T13:51:01.481 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114914.log.gz 2024-09-16T13:51:01.481 INFO:teuthology.orchestra.run.smithi172.stderr: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.121705.log.gz 2024-09-16T13:51:01.481 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.115925.log.gz 2024-09-16T13:51:01.481 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.121157.log.gz 2024-09-16T13:51:01.481 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69686.log.gz 2024-09-16T13:51:01.481 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58228.log.gz 2024-09-16T13:51:01.481 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.50320.log.gz 2024-09-16T13:51:01.481 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.118196.log.gz 2024-09-16T13:51:01.481 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135803.log.gz 2024-09-16T13:51:01.481 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138700.log.gz 2024-09-16T13:51:01.481 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116623.log.gz 2024-09-16T13:51:01.481 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126285.log.gz 2024-09-16T13:51:01.481 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137483.log.gz 2024-09-16T13:51:01.482 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.88785.log.gz 2024-09-16T13:51:01.482 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73470.log.gz 2024-09-16T13:51:01.482 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.120847.log.gz 2024-09-16T13:51:01.482 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133448.log.gz 2024-09-16T13:51:01.482 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.95120.log.gz 2024-09-16T13:51:01.482 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67474.log.gz 2024-09-16T13:51:01.482 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.95333.log.gz 2024-09-16T13:51:01.482 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115401.log.gz 2024-09-16T13:51:01.482 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38454.log.gz 2024-09-16T13:51:01.482 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.132653.log.gz 2024-09-16T13:51:01.489 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% 57.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63540.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.27057.log.gz 2024-09-16T13:51:01.489 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.489 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65315.log.gz 2024-09-16T13:51:01.489 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.105040.log.gz 2024-09-16T13:51:01.489 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.114811.log.gz 2024-09-16T13:51:01.489 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116878.log.gz 2024-09-16T13:51:01.489 INFO:teuthology.orchestra.run.smithi172.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.63502.log.gz 2024-09-16T13:51:01.489 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.141414.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.66363.log.gz 2024-09-16T13:51:01.489 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.489 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46220.log.gz 2024-09-16T13:51:01.489 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114242.log.gz 2024-09-16T13:51:01.489 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114287.log.gz 2024-09-16T13:51:01.490 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.60308.log.gz 2024-09-16T13:51:01.490 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.78089.log.gz 2024-09-16T13:51:01.490 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91945.log.gz 2024-09-16T13:51:01.490 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70437.log.gz 2024-09-16T13:51:01.490 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91736.log.gz 2024-09-16T13:51:01.490 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51553.log.gz 2024-09-16T13:51:01.490 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58462.log.gz 2024-09-16T13:51:01.490 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.122361.log.gz 2024-09-16T13:51:01.490 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.74175.log.gz 2024-09-16T13:51:01.490 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.59990.log.gz 2024-09-16T13:51:01.490 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.91649.log.gz 2024-09-16T13:51:01.490 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.136643.log.gz 2024-09-16T13:51:01.490 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92747.log.gz 2024-09-16T13:51:01.490 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79116.log.gz 2024-09-16T13:51:01.491 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.82088.log.gz 2024-09-16T13:51:01.491 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.100199.log.gz 2024-09-16T13:51:01.491 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.28126.log.gz 2024-09-16T13:51:01.491 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45277.log.gz 89.9% 2024-09-16T13:51:01.491 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.138027.log.gz 2024-09-16T13:51:01.491 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65268.log.gz 2024-09-16T13:51:01.491 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76590.log.gz 2024-09-16T13:51:01.491 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.72047.log.gz 2024-09-16T13:51:01.491 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91776.log.gz 2024-09-16T13:51:01.491 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137163.log.gz 2024-09-16T13:51:01.491 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44282.log.gz 2024-09-16T13:51:01.491 INFO:teuthology.orchestra.run.smithi172.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.96772.log.gz 2024-09-16T13:51:01.492 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.73812.log.gz 2024-09-16T13:51:01.492 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68974.log.gz 2024-09-16T13:51:01.492 INFO:teuthology.orchestra.run.smithi172.stderr: 92.2% 89.8% -- replaced with /var/log/ceph/cluster1-client.admin.118088.log.gz 2024-09-16T13:51:01.492 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.100282.log.gz 2024-09-16T13:51:01.492 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.125379.log.gz 2024-09-16T13:51:01.492 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.28487.log.gz 2024-09-16T13:51:01.492 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116204.log.gz 2024-09-16T13:51:01.492 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113258.log.gz 2024-09-16T13:51:01.492 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125107.log.gz 2024-09-16T13:51:01.492 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126664.log.gz 2024-09-16T13:51:01.492 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46927.log.gz 2024-09-16T13:51:01.492 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.39508.log.gz 2024-09-16T13:51:01.492 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.28283.log.gz 2024-09-16T13:51:01.493 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76835.log.gz 2024-09-16T13:51:01.493 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.119854.log.gz 2024-09-16T13:51:01.493 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46971.log.gz 2024-09-16T13:51:01.493 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55641.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.69520.log.gz 2024-09-16T13:51:01.493 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.493 INFO:teuthology.orchestra.run.smithi172.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.119793.log.gz 2024-09-16T13:51:01.493 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% 89.7% -- replaced with /var/log/ceph/cluster1-client.admin.122181.log.gz 2024-09-16T13:51:01.493 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.27628.log.gz 2024-09-16T13:51:01.493 INFO:teuthology.orchestra.run.smithi172.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.142358.log.gz 2024-09-16T13:51:01.493 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.39984.log.gz 2024-09-16T13:51:01.493 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.107827.log.gz 2024-09-16T13:51:01.493 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109387.log.gz 2024-09-16T13:51:01.493 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91315.log.gz 2024-09-16T13:51:01.494 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63893.log.gz 2024-09-16T13:51:01.494 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112520.log.gz 2024-09-16T13:51:01.494 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.47252.log.gz 2024-09-16T13:51:01.494 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135861.log.gz 2024-09-16T13:51:01.494 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45339.log.gz 2024-09-16T13:51:01.494 INFO:teuthology.orchestra.run.smithi172.stderr: 67.1% -- replaced with /var/log/ceph/cluster1-client.mirror.108724.log.gz 2024-09-16T13:51:01.494 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135189.log.gz 2024-09-16T13:51:01.494 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101235.log.gz 2024-09-16T13:51:01.494 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.38609.log.gz 2024-09-16T13:51:01.494 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34790.log.gz 2024-09-16T13:51:01.494 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125108.log.gz 2024-09-16T13:51:01.494 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125190.log.gz 2024-09-16T13:51:01.494 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.109833.log.gz 2024-09-16T13:51:01.495 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.admin.86515.log.gz 2024-09-16T13:51:01.495 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.97858.log.gz 2024-09-16T13:51:01.495 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69188.log.gz 2024-09-16T13:51:01.495 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% 90.0% -- replaced with /var/log/ceph/cluster2-client.admin.37069.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.42960.log.gz 2024-09-16T13:51:01.495 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.495 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.73853.log.gz 2024-09-16T13:51:01.495 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44113.log.gz 2024-09-16T13:51:01.495 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90263.log.gz 2024-09-16T13:51:01.495 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81729.log.gz 2024-09-16T13:51:01.495 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57179.log.gz 2024-09-16T13:51:01.495 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44283.log.gz 2024-09-16T13:51:01.495 INFO:teuthology.orchestra.run.smithi172.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.64073.log.gz 2024-09-16T13:51:01.495 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.92324.log.gz 2024-09-16T13:51:01.495 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34213.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.122772.log.gz 2024-09-16T13:51:01.496 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.496 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.86629.log.gz 2024-09-16T13:51:01.496 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53334.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.80462.log.gz 2024-09-16T13:51:01.496 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.496 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.34136.log.gz 2024-09-16T13:51:01.496 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.140149.log.gz 2024-09-16T13:51:01.496 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.125235.log.gz 2024-09-16T13:51:01.496 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81559.log.gz 2024-09-16T13:51:01.496 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.33415.log.gz 2024-09-16T13:51:01.496 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78066.log.gz 2024-09-16T13:51:01.496 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46928.log.gz 2024-09-16T13:51:01.496 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103549.log.gz 2024-09-16T13:51:01.496 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.85009.log.gz 2024-09-16T13:51:01.497 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32242.log.gz 2024-09-16T13:51:01.497 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.117806.log.gz 2024-09-16T13:51:01.497 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.84026.log.gz 2024-09-16T13:51:01.497 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101210.log.gz 2024-09-16T13:51:01.497 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135591.log.gz 2024-09-16T13:51:01.497 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33622.log.gz 2024-09-16T13:51:01.497 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.96947.log.gz 2024-09-16T13:51:01.497 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.106685.log.gz 2024-09-16T13:51:01.497 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.124604.log.gz 2024-09-16T13:51:01.497 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53543.log.gz 2024-09-16T13:51:01.497 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.113108.log.gz 2024-09-16T13:51:01.497 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.47899.log.gz 2024-09-16T13:51:01.497 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101446.log.gz 2024-09-16T13:51:01.498 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124480.log.gz 2024-09-16T13:51:01.498 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42182.log.gz 2024-09-16T13:51:01.525 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134109.log.gz 89.9% 2024-09-16T13:51:01.525 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.134154.log.gz 2024-09-16T13:51:01.525 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% 89.9% -- replaced with /var/log/ceph/cluster2-client.admin.84379.log.gz 2024-09-16T13:51:01.525 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.76506.log.gz 2024-09-16T13:51:01.525 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56901.log.gz 2024-09-16T13:51:01.525 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102913.log.gz 2024-09-16T13:51:01.525 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116203.log.gz 2024-09-16T13:51:01.525 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115129.log.gz 2024-09-16T13:51:01.526 INFO:teuthology.orchestra.run.smithi172.stderr: 66.6% -- replaced with /var/log/ceph/cluster1-client.mirror.86582.log.gz 2024-09-16T13:51:01.526 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% 89.9% -- replaced with /var/log/ceph/cluster2-client.admin.107255.log.gz 2024-09-16T13:51:01.526 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.64652.log.gz 2024-09-16T13:51:01.526 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.100602.log.gz 2024-09-16T13:51:01.526 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.108115.log.gz 2024-09-16T13:51:01.526 INFO:teuthology.orchestra.run.smithi172.stderr: 19.6% -- replaced with /var/log/ceph/cluster1-client.admin.132337.log.gz 2024-09-16T13:51:01.526 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.84381.log.gz 2024-09-16T13:51:01.526 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74536.log.gz 2024-09-16T13:51:01.526 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124749.log.gz 2024-09-16T13:51:01.526 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102994.log.gz 2024-09-16T13:51:01.526 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.125740.log.gz 2024-09-16T13:51:01.526 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.85002.log.gz 2024-09-16T13:51:01.526 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91607.log.gz 2024-09-16T13:51:01.526 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.108543.log.gz 2024-09-16T13:51:01.527 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126033.log.gz 2024-09-16T13:51:01.527 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53730.log.gz 2024-09-16T13:51:01.527 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65794.log.gz 2024-09-16T13:51:01.527 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116139.log.gz 2024-09-16T13:51:01.527 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.103438.log.gz 2024-09-16T13:51:01.527 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.113301.log.gz 2024-09-16T13:51:01.527 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138109.log.gz 2024-09-16T13:51:01.527 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.91524.log.gz 2024-09-16T13:51:01.527 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125444.log.gz 2024-09-16T13:51:01.527 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.91820.log.gz 2024-09-16T13:51:01.527 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.39325.log.gz 2024-09-16T13:51:01.527 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41802.log.gz 2024-09-16T13:51:01.527 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62956.log.gz 2024-09-16T13:51:01.528 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53605.log.gz 2024-09-16T13:51:01.528 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137902.log.gz 2024-09-16T13:51:01.528 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137398.log.gz 2024-09-16T13:51:01.528 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.122669.log.gz 2024-09-16T13:51:01.528 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.90264.log.gz 2024-09-16T13:51:01.528 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.140015.log.gz 2024-09-16T13:51:01.528 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89425.log.gz 2024-09-16T13:51:01.528 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56585.log.gz 2024-09-16T13:51:01.528 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70037.log.gz 2024-09-16T13:51:01.528 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.28369.log.gz 2024-09-16T13:51:01.528 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53816.log.gz 2024-09-16T13:51:01.528 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125526.log.gz 2024-09-16T13:51:01.528 INFO:teuthology.orchestra.run.smithi172.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.admin.142366.log.gz 2024-09-16T13:51:01.529 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43380.log.gz 2024-09-16T13:51:01.529 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56402.log.gz 2024-09-16T13:51:01.529 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.59989.log.gz 2024-09-16T13:51:01.529 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50024.log.gz 2024-09-16T13:51:01.529 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.48304.log.gz 2024-09-16T13:51:01.529 INFO:teuthology.orchestra.run.smithi172.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74248.log.gz 2024-09-16T13:51:01.529 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.60525.log.gz 2024-09-16T13:51:01.529 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.73652.log.gz 2024-09-16T13:51:01.529 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29970.log.gz 2024-09-16T13:51:01.529 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138321.log.gz 2024-09-16T13:51:01.529 INFO:teuthology.orchestra.run.smithi172.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-client.admin.38750.log.gz 2024-09-16T13:51:01.529 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.71899.log.gz 2024-09-16T13:51:01.530 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.71975.log.gz 2024-09-16T13:51:01.530 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.106985.log.gz 2024-09-16T13:51:01.530 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.27333.log.gz 2024-09-16T13:51:01.530 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.95214.log.gz 2024-09-16T13:51:01.530 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58227.log.gz 2024-09-16T13:51:01.530 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.101314.log.gz 2024-09-16T13:51:01.530 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.131346.log.gz 2024-09-16T13:51:01.530 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% 96.3% -- replaced with /var/log/ceph/cluster1-client.mirror.29883.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.49910.log.gz 2024-09-16T13:51:01.530 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.530 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.108817.log.gz 2024-09-16T13:51:01.530 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67579.log.gz 2024-09-16T13:51:01.530 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.80886.log.gz 2024-09-16T13:51:01.530 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81351.log.gz 90.0% 2024-09-16T13:51:01.530 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.137373.log.gz 2024-09-16T13:51:01.531 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.132526.log.gz 2024-09-16T13:51:01.531 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% 89.5% -- replaced with /var/log/ceph/cluster2-client.admin.130403.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.130817.log.gz 2024-09-16T13:51:01.531 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.531 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80929.log.gz 2024-09-16T13:51:01.531 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.67918.log.gz 2024-09-16T13:51:01.531 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29925.log.gz 2024-09-16T13:51:01.531 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.129938.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.127929.log.gz 2024-09-16T13:51:01.531 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.531 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster2-client.admin.37941.log.gz 2024-09-16T13:51:01.531 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68209.log.gz 2024-09-16T13:51:01.531 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41337.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.33924.log.gz 2024-09-16T13:51:01.531 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.531 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.97342.log.gz 2024-09-16T13:51:01.531 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.140881.log.gz 2024-09-16T13:51:01.532 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.54905.log.gz 2024-09-16T13:51:01.532 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56062.log.gz 2024-09-16T13:51:01.532 INFO:teuthology.orchestra.run.smithi172.stderr: 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.85784.log.gz 2024-09-16T13:51:01.532 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.34002.log.gz 2024-09-16T13:51:01.532 INFO:teuthology.orchestra.run.smithi172.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85857.log.gz 2024-09-16T13:51:01.532 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.106867.log.gz 2024-09-16T13:51:01.532 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75719.log.gz 2024-09-16T13:51:01.532 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33714.log.gz 89.8% 2024-09-16T13:51:01.532 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.114349.log.gz 2024-09-16T13:51:01.532 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.79771.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.111257.log.gz 2024-09-16T13:51:01.532 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.532 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.61509.log.gz 2024-09-16T13:51:01.532 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68783.log.gz 2024-09-16T13:51:01.533 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136346.log.gz 2024-09-16T13:51:01.533 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77100.log.gz 2024-09-16T13:51:01.533 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.33835.log.gz 2024-09-16T13:51:01.533 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.114452.log.gz 2024-09-16T13:51:01.533 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.76945.log.gz 2024-09-16T13:51:01.533 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48875.log.gz 2024-09-16T13:51:01.533 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77251.log.gz 2024-09-16T13:51:01.533 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57745.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.45127.log.gz 2024-09-16T13:51:01.533 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.533 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53088.log.gz 2024-09-16T13:51:01.533 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126793.log.gz 2024-09-16T13:51:01.533 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44641.log.gz 2024-09-16T13:51:01.533 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65102.log.gz 2024-09-16T13:51:01.533 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.74450.log.gz 2024-09-16T13:51:01.534 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44640.log.gz 2024-09-16T13:51:01.534 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61603.log.gz 2024-09-16T13:51:01.534 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster1-client.admin.109487.log.gz 2024-09-16T13:51:01.534 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43594.log.gz 2024-09-16T13:51:01.534 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111368.log.gz 2024-09-16T13:51:01.534 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55535.log.gz 2024-09-16T13:51:01.534 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53962.log.gz 2024-09-16T13:51:01.534 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.125864.log.gz 2024-09-16T13:51:01.534 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44596.log.gz 2.3% 2024-09-16T13:51:01.534 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.26816.log.gz 2024-09-16T13:51:01.534 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.36071.log.gz 2024-09-16T13:51:01.534 INFO:teuthology.orchestra.run.smithi172.stderr: 80.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62758.log.gz 2024-09-16T13:51:01.534 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64891.log.gz 2024-09-16T13:51:01.534 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.104048.log.gz 2024-09-16T13:51:01.535 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46331.log.gz 2024-09-16T13:51:01.535 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.85804.log.gz 2024-09-16T13:51:01.535 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42225.log.gz 2024-09-16T13:51:01.535 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.142008.log.gz 2024-09-16T13:51:01.535 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.99157.log.gz 2024-09-16T13:51:01.535 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.118844.log.gz 2024-09-16T13:51:01.535 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135654.log.gz 2024-09-16T13:51:01.535 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.142013.log.gz 2024-09-16T13:51:01.535 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112249.log.gz 2024-09-16T13:51:01.535 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.123196.log.gz 2024-09-16T13:51:01.535 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.142328.log.gz 2024-09-16T13:51:01.535 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.129941.log.gz 2024-09-16T13:51:01.535 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.89644.log.gz 2024-09-16T13:51:01.535 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34684.log.gz 2024-09-16T13:51:01.535 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.121838.log.gz 2024-09-16T13:51:01.536 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.75694.log.gz 2024-09-16T13:51:01.536 INFO:teuthology.orchestra.run.smithi172.stderr: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.86219.log.gz 2024-09-16T13:51:01.536 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55980.log.gz 2024-09-16T13:51:01.536 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56188.log.gz 2024-09-16T13:51:01.536 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123069.log.gz 2024-09-16T13:51:01.536 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.77564.log.gz 2024-09-16T13:51:01.536 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.95780.log.gz 2024-09-16T13:51:01.536 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61717.log.gz 2024-09-16T13:51:01.536 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112940.log.gz 2024-09-16T13:51:01.536 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.107050.log.gz 2024-09-16T13:51:01.536 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74844.log.gz 2024-09-16T13:51:01.536 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74645.log.gz 2024-09-16T13:51:01.536 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.139375.log.gz 2024-09-16T13:51:01.536 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127400.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.52597.log.gz 2024-09-16T13:51:01.537 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.537 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133183.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.112881.log.gz 2024-09-16T13:51:01.537 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.537 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.112879.log.gz 2024-09-16T13:51:01.537 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.142011.log.gz 2024-09-16T13:51:01.537 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58293.log.gz 2024-09-16T13:51:01.537 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.36483.log.gz 2024-09-16T13:51:01.537 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45442.log.gz 2024-09-16T13:51:01.537 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster1-client.admin.61650.log.gz 2024-09-16T13:51:01.537 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80930.log.gz 2024-09-16T13:51:01.537 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74268.log.gz 2024-09-16T13:51:01.537 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32740.log.gz 2024-09-16T13:51:01.537 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.84562.log.gz 2024-09-16T13:51:01.537 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102890.log.gz 2024-09-16T13:51:01.538 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.76385.log.gz 2024-09-16T13:51:01.538 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.110186.log.gz 2024-09-16T13:51:01.538 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.73538.log.gz 2024-09-16T13:51:01.538 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% 80.2% -- replaced with /var/log/ceph/cluster1-client.admin.71274.log.gz 2024-09-16T13:51:01.538 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.87426.log.gz 2024-09-16T13:51:01.538 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.128055.log.gz 2024-09-16T13:51:01.538 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.127004.log.gz 2024-09-16T13:51:01.538 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119585.log.gz 2024-09-16T13:51:01.538 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.135130.log.gz 2024-09-16T13:51:01.538 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64977.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.72867.log.gz 2024-09-16T13:51:01.538 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.538 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103416.log.gz 2024-09-16T13:51:01.538 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111192.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.102640.log.gz 2024-09-16T13:51:01.538 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.538 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93294.log.gz 2024-09-16T13:51:01.539 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114939.log.gz 2024-09-16T13:51:01.539 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124981.log.gz 2024-09-16T13:51:01.539 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.74382.log.gz 2024-09-16T13:51:01.539 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.98034.log.gz 2024-09-16T13:51:01.539 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.103060.log.gz 2024-09-16T13:51:01.539 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.84524.log.gz 2024-09-16T13:51:01.539 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62608.log.gz 2024-09-16T13:51:01.539 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85649.log.gz 2024-09-16T13:51:01.539 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.59552.log.gz 2024-09-16T13:51:01.539 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44013.log.gz 2024-09-16T13:51:01.539 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33838.log.gz 2024-09-16T13:51:01.539 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.51442.log.gz 2024-09-16T13:51:01.539 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53855.log.gz 2024-09-16T13:51:01.539 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32109.log.gz 2024-09-16T13:51:01.540 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.36834.log.gz 2024-09-16T13:51:01.540 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33200.log.gz 2024-09-16T13:51:01.540 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.114771.log.gz 2024-09-16T13:51:01.540 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93168.log.gz 2024-09-16T13:51:01.540 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81137.log.gz 2024-09-16T13:51:01.540 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.48705.log.gz 2024-09-16T13:51:01.540 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53248.log.gz 2024-09-16T13:51:01.540 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.141872.log.gz 2024-09-16T13:51:01.540 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53520.log.gz 2024-09-16T13:51:01.540 INFO:teuthology.orchestra.run.smithi172.stderr: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50802.log.gz 2024-09-16T13:51:01.541 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.102639.log.gz 2024-09-16T13:51:01.541 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105865.log.gz 2024-09-16T13:51:01.541 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68469.log.gz 2024-09-16T13:51:01.541 INFO:teuthology.orchestra.run.smithi172.stderr: 66.8% -- replaced with /var/log/ceph/cluster1-client.mirror.97383.log.gz 2024-09-16T13:51:01.541 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30219.log.gz 2024-09-16T13:51:01.541 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.29757.log.gz 2024-09-16T13:51:01.541 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127930.log.gz 2024-09-16T13:51:01.541 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137270.log.gz 2024-09-16T13:51:01.541 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.107173.log.gz 2024-09-16T13:51:01.541 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.137541.log.gz 2024-09-16T13:51:01.541 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68530.log.gz 2024-09-16T13:51:01.541 INFO:teuthology.orchestra.run.smithi172.stderr: 91.4% -- replaced with /var/log/ceph/cluster1-client.admin.97316.log.gz 2024-09-16T13:51:01.541 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.95248.log.gz 2024-09-16T13:51:01.541 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38225.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.75802.log.gz 2024-09-16T13:51:01.542 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.542 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.129537.log.gz 2024-09-16T13:51:01.542 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105760.log.gz 2024-09-16T13:51:01.542 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30351.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.34043.log.gz 2024-09-16T13:51:01.542 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.542 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58059.log.gz 89.9% 2024-09-16T13:51:01.542 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.66366.log.gz 2024-09-16T13:51:01.542 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65313.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.78931.log.gz 2024-09-16T13:51:01.542 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.542 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139623.log.gz 2024-09-16T13:51:01.542 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139775.log.gz 2024-09-16T13:51:01.542 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64527.log.gz 2024-09-16T13:51:01.542 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51419.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.72676.log.gz 2024-09-16T13:51:01.542 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.543 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64739.log.gz 2024-09-16T13:51:01.543 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136220.log.gz 2024-09-16T13:51:01.543 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136281.log.gz 2024-09-16T13:51:01.543 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136323.log.gz 2024-09-16T13:51:01.543 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112982.log.gz 2024-09-16T13:51:01.543 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.101422.log.gz 2024-09-16T13:51:01.543 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.108657.log.gz 2024-09-16T13:51:01.543 INFO:teuthology.orchestra.run.smithi172.stderr: 67.1% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74517.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.127845.log.gz 2024-09-16T13:51:01.543 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.543 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.98595.log.gz 2024-09-16T13:51:01.543 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.admin.130049.log.gz 2024-09-16T13:51:01.543 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.133728.log.gz 2024-09-16T13:51:01.543 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66153.log.gz 90.0% 2024-09-16T13:51:01.543 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.102917.log.gz 2024-09-16T13:51:01.544 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.68347.log.gz 2024-09-16T13:51:01.544 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.60991.log.gz 2024-09-16T13:51:01.544 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79031.log.gz 2024-09-16T13:51:01.544 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.60769.log.gz 2024-09-16T13:51:01.544 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.51036.log.gz 2024-09-16T13:51:01.544 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62494.log.gz 2024-09-16T13:51:01.544 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.135398.log.gz 2024-09-16T13:51:01.544 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116854.log.gz 2024-09-16T13:51:01.544 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30933.log.gz 2024-09-16T13:51:01.544 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46541.log.gz 2024-09-16T13:51:01.544 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.47253.log.gz 2024-09-16T13:51:01.544 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56611.log.gz 2024-09-16T13:51:01.544 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86675.log.gz 2024-09-16T13:51:01.545 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.84710.log.gz 2024-09-16T13:51:01.545 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101447.log.gz 2024-09-16T13:51:01.545 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111614.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.116688.log.gz 2024-09-16T13:51:01.545 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.545 INFO:teuthology.orchestra.run.smithi172.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.108031.log.gz 2024-09-16T13:51:01.545 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65942.log.gz 2024-09-16T13:51:01.545 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103437.log.gz 2024-09-16T13:51:01.545 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81223.log.gz 2024-09-16T13:51:01.545 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.92199.log.gz 2024-09-16T13:51:01.545 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.32030.log.gz 2024-09-16T13:51:01.545 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.112521.log.gz 2024-09-16T13:51:01.545 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.131647.log.gz 2024-09-16T13:51:01.545 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46391.log.gz 2024-09-16T13:51:01.545 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93461.log.gz 2024-09-16T13:51:01.545 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89718.log.gz 2024-09-16T13:51:01.546 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30557.log.gz 2024-09-16T13:51:01.546 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89131.log.gz 2024-09-16T13:51:01.546 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.86105.log.gz 2024-09-16T13:51:01.546 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.139122.log.gz 2024-09-16T13:51:01.546 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.95820.log.gz 66.6% 2024-09-16T13:51:01.546 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.62173.log.gz 2024-09-16T13:51:01.546 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.139475.log.gz 2024-09-16T13:51:01.546 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.141412.log.gz 2024-09-16T13:51:01.546 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.131936.log.gz 2024-09-16T13:51:01.546 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.116646.log.gz 2024-09-16T13:51:01.546 INFO:teuthology.orchestra.run.smithi172.stderr: 67.2% -- replaced with /var/log/ceph/cluster1-client.admin.26765.log.gz 2024-09-16T13:51:01.546 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70131.log.gz 2024-09-16T13:51:01.546 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.62106.log.gz 89.8% 2024-09-16T13:51:01.546 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.45909.log.gz 2024-09-16T13:51:01.547 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63076.log.gz 2024-09-16T13:51:01.547 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67202.log.gz 2024-09-16T13:51:01.547 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.90182.log.gz 2024-09-16T13:51:01.547 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43756.log.gz 2024-09-16T13:51:01.547 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.109159.log.gz 2024-09-16T13:51:01.547 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.121904.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.123426.log.gz 2024-09-16T13:51:01.547 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.547 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.97918.log.gz 2024-09-16T13:51:01.547 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30093.log.gz 2024-09-16T13:51:01.547 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54385.log.gz 2024-09-16T13:51:01.547 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.137606.log.gz 2024-09-16T13:51:01.547 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123699.log.gz 2024-09-16T13:51:01.547 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.88280.log.gz 2024-09-16T13:51:01.547 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.97818.log.gz 2024-09-16T13:51:01.548 INFO:teuthology.orchestra.run.smithi172.stderr: 92.2% 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.72384.log.gz 2024-09-16T13:51:01.548 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.35020.log.gz 2024-09-16T13:51:01.548 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.111366.log.gz 2024-09-16T13:51:01.548 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.118336.log.gz 2024-09-16T13:51:01.548 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80193.log.gz 2024-09-16T13:51:01.548 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 67.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56274.log.gz 2024-09-16T13:51:01.548 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.38132.log.gz 2024-09-16T13:51:01.548 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51978.log.gz 2024-09-16T13:51:01.548 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44914.log.gz 2024-09-16T13:51:01.548 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.110227.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.65817.log.gz 2024-09-16T13:51:01.548 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.548 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.98700.log.gz 2024-09-16T13:51:01.548 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.98857.log.gz 2024-09-16T13:51:01.549 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.124561.log.gz 2024-09-16T13:51:01.549 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115506.log.gz 2024-09-16T13:51:01.549 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.84570.log.gz 2024-09-16T13:51:01.549 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.95526.log.gz 2024-09-16T13:51:01.549 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.96129.log.gz 2024-09-16T13:51:01.549 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134387.log.gz 2024-09-16T13:51:01.549 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.103758.log.gz 2024-09-16T13:51:01.549 INFO:teuthology.orchestra.run.smithi172.stderr: 17.4% -- replaced with /var/log/ceph/cluster2-client.admin.85077.log.gz 2024-09-16T13:51:01.549 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40561.log.gz 2024-09-16T13:51:01.549 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% -- replaced with /var/log/ceph/cluster1-client.admin.73766.log.gz 2024-09-16T13:51:01.549 INFO:teuthology.orchestra.run.smithi172.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.admin.73625.log.gz 2024-09-16T13:51:01.549 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.93252.log.gz 2024-09-16T13:51:01.549 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93967.log.gz 2024-09-16T13:51:01.549 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38568.log.gz 2024-09-16T13:51:01.550 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.76568.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.37524.log.gz 2024-09-16T13:51:01.550 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.550 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.136073.log.gz 2024-09-16T13:51:01.550 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138214.log.gz 2024-09-16T13:51:01.550 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64526.log.gz 2024-09-16T13:51:01.550 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.97248.log.gz 2024-09-16T13:51:01.550 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54781.log.gz 2024-09-16T13:51:01.550 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57600.log.gz 2024-09-16T13:51:01.550 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.60314.log.gz 89.9% 2024-09-16T13:51:01.550 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.32949.log.gz 2024-09-16T13:51:01.550 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.131533.log.gz 2024-09-16T13:51:01.550 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.102301.log.gz 2024-09-16T13:51:01.550 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.56336.log.gz 2024-09-16T13:51:01.550 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.61074.log.gz 2024-09-16T13:51:01.551 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.126663.log.gz 2024-09-16T13:51:01.551 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 3.5% -- replaced with /var/log/ceph/cluster1-client.mirror.88742.log.gz 2024-09-16T13:51:01.551 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.26799.log.gz 2024-09-16T13:51:01.551 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.85717.log.gz 2024-09-16T13:51:01.551 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.48554.log.gz 2024-09-16T13:51:01.551 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53438.log.gz 2024-09-16T13:51:01.551 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44493.log.gz 2024-09-16T13:51:01.551 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.92072.log.gz 2024-09-16T13:51:01.551 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 18.5% -- replaced with /var/log/ceph/cluster1-client.mirror.111562.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.109940.log.gz 2024-09-16T13:51:01.551 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.551 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.80087.log.gz 2024-09-16T13:51:01.551 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100471.log.gz 2024-09-16T13:51:01.551 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.140882.log.gz 2024-09-16T13:51:01.551 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.49504.log.gz 2024-09-16T13:51:01.552 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.32950.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.118745.log.gz 2024-09-16T13:51:01.552 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.552 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 80.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54990.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.51088.log.gz 2024-09-16T13:51:01.552 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.552 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.134878.log.gz 2024-09-16T13:51:01.552 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66239.log.gz 2024-09-16T13:51:01.552 INFO:teuthology.orchestra.run.smithi172.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.119565.log.gz 2024-09-16T13:51:01.552 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.68210.log.gz 2024-09-16T13:51:01.552 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.89905.log.gz 2024-09-16T13:51:01.552 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.81247.log.gz 2024-09-16T13:51:01.552 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.101101.log.gz 2024-09-16T13:51:01.552 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.33923.log.gz 2024-09-16T13:51:01.552 INFO:teuthology.orchestra.run.smithi172.stderr: 88.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.105019.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.75835.log.gz 2024-09-16T13:51:01.552 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.553 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.101525.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.72354.log.gz 2024-09-16T13:51:01.553 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.553 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.36068.log.gz 2024-09-16T13:51:01.553 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.60311.log.gz 2024-09-16T13:51:01.553 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.106546.log.gz 2024-09-16T13:51:01.553 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116374.log.gz 2024-09-16T13:51:01.553 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57874.log.gz 2024-09-16T13:51:01.553 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% 96.3% -- replaced with /var/log/ceph/cluster1-client.admin.50293.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.120497.log.gz 2024-09-16T13:51:01.553 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.553 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38111.log.gz 2024-09-16T13:51:01.553 INFO:teuthology.orchestra.run.smithi172.stderr: 19.0% -- replaced with /var/log/ceph/cluster1-client.admin.110817.log.gz 2024-09-16T13:51:01.553 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87752.log.gz 2024-09-16T13:51:01.553 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79537.log.gz 2024-09-16T13:51:01.553 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.136809.log.gz 2024-09-16T13:51:01.554 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.103481.log.gz 2024-09-16T13:51:01.554 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44533.log.gz 2024-09-16T13:51:01.554 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.admin.84563.log.gz 2024-09-16T13:51:01.554 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.97134.log.gz 2024-09-16T13:51:01.554 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57452.log.gz 2024-09-16T13:51:01.554 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.141331.log.gz 2024-09-16T13:51:01.554 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.124058.log.gz 2024-09-16T13:51:01.554 INFO:teuthology.orchestra.run.smithi172.stderr: 79.9% -- replaced with /var/log/ceph/cluster1-client.mirror.133847.log.gz 2024-09-16T13:51:01.554 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85763.log.gz 2024-09-16T13:51:01.554 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.84792.log.gz 2024-09-16T13:51:01.554 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.110856.log.gz 2024-09-16T13:51:01.554 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.30561.log.gz 2024-09-16T13:51:01.554 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.138951.log.gz 2024-09-16T13:51:01.554 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45590.log.gz 2024-09-16T13:51:01.555 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.123099.log.gz 2024-09-16T13:51:01.555 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64396.log.gz 2024-09-16T13:51:01.555 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.118738.log.gz 2024-09-16T13:51:01.555 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.114307.log.gz 2024-09-16T13:51:01.555 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.129942.log.gz 2024-09-16T13:51:01.555 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57032.log.gz 2024-09-16T13:51:01.555 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108797.log.gz 2024-09-16T13:51:01.555 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74978.log.gz 2024-09-16T13:51:01.555 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.81621.log.gz 2024-09-16T13:51:01.555 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.95983.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.28489.log.gz 2024-09-16T13:51:01.555 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.555 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.141332.log.gz 2024-09-16T13:51:01.555 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.116082.log.gz 2024-09-16T13:51:01.555 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.84223.log.gz 2024-09-16T13:51:01.556 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123134.log.gz 2024-09-16T13:51:01.556 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.97476.log.gz 2024-09-16T13:51:01.556 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.85945.log.gz 2024-09-16T13:51:01.556 INFO:teuthology.orchestra.run.smithi172.stderr: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.108475.log.gz 2024-09-16T13:51:01.556 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70494.log.gz 2024-09-16T13:51:01.556 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.88333.log.gz 2024-09-16T13:51:01.556 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57092.log.gz 2024-09-16T13:51:01.556 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.126078.log.gz 2024-09-16T13:51:01.556 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55643.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.42915.log.gz 2024-09-16T13:51:01.556 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.556 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58168.log.gz 2024-09-16T13:51:01.556 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% 90.0% -- replaced with /var/log/ceph/cluster2-client.admin.142191.log.gz 2024-09-16T13:51:01.556 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.41293.log.gz 2024-09-16T13:51:01.557 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55348.log.gz 2024-09-16T13:51:01.557 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.111113.log.gz 2024-09-16T13:51:01.557 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.30679.log.gz 2024-09-16T13:51:01.557 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.29969.log.gz 2024-09-16T13:51:01.557 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.114407.log.gz 2024-09-16T13:51:01.557 INFO:teuthology.orchestra.run.smithi172.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.119679.log.gz 2024-09-16T13:51:01.557 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69895.log.gz 2024-09-16T13:51:01.557 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57303.log.gz 2024-09-16T13:51:01.557 INFO:teuthology.orchestra.run.smithi172.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.61790.log.gz 2024-09-16T13:51:01.557 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53461.log.gz 2024-09-16T13:51:01.557 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.71455.log.gz 2024-09-16T13:51:01.557 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.104488.log.gz 2024-09-16T13:51:01.557 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% 18.5% -- replaced with /var/log/ceph/cluster1-client.admin.118086.log.gz 2024-09-16T13:51:01.557 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.132613.log.gz 2024-09-16T13:51:01.558 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% 80.2% -- replaced with /var/log/ceph/cluster1-client.admin.48282.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.40188.log.gz 2024-09-16T13:51:01.558 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.558 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54300.log.gz 2024-09-16T13:51:01.558 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.131822.log.gz 2024-09-16T13:51:01.558 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.94460.log.gz 2024-09-16T13:51:01.558 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.33199.log.gz 2024-09-16T13:51:01.558 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.141592.log.gz 2024-09-16T13:51:01.558 INFO:teuthology.orchestra.run.smithi172.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.36293.log.gz 2024-09-16T13:51:01.558 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67894.log.gz 2024-09-16T13:51:01.558 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.49616.log.gz 2024-09-16T13:51:01.558 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.115655.log.gz 2024-09-16T13:51:01.558 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.90557.log.gz 2024-09-16T13:51:01.558 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127634.log.gz 2024-09-16T13:51:01.558 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.73072.log.gz 80.1% 2024-09-16T13:51:01.559 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.98267.log.gz 2024-09-16T13:51:01.559 INFO:teuthology.orchestra.run.smithi172.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-client.admin.62220.log.gz 2024-09-16T13:51:01.559 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.127695.log.gz 2024-09-16T13:51:01.559 INFO:teuthology.orchestra.run.smithi172.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.admin.142157.log.gz 2024-09-16T13:51:01.559 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31400.log.gz 2024-09-16T13:51:01.559 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.99646.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.111502.log.gz 2024-09-16T13:51:01.559 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.559 INFO:teuthology.orchestra.run.smithi172.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39178.log.gz 2024-09-16T13:51:01.559 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56124.log.gz 2024-09-16T13:51:01.559 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.123614.log.gz 2024-09-16T13:51:01.559 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31820.log.gz 2024-09-16T13:51:01.559 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.87398.log.gz 2024-09-16T13:51:01.559 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55705.log.gz 2024-09-16T13:51:01.559 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.106966.log.gz 89.5% 2024-09-16T13:51:01.559 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.130857.log.gz 2024-09-16T13:51:01.560 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.31730.log.gz 2024-09-16T13:51:01.560 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.88172.log.gz 2024-09-16T13:51:01.560 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% 90.0% -- replaced with /var/log/ceph/cluster1-client.admin.73424.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.105501.log.gz 2024-09-16T13:51:01.560 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T13:51:01.566 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster1-client.admin.74732.log.gz 2024-09-16T13:51:02.118 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.134326.log.gz 2024-09-16T13:51:02.133 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.93002.log.gz 2024-09-16T13:51:02.136 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41552.log.gz 2024-09-16T13:51:02.139 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.97590.log.gz 2024-09-16T13:51:02.386 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55076.log.gz 2024-09-16T13:51:07.898 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.113889.log.gz 2024-09-16T13:51:12.117 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.admin.129901.log.gz 2024-09-16T13:51:12.118 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.138742.log.gz 2024-09-16T13:51:12.118 INFO:teuthology.orchestra.run.smithi172.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.admin.84377.log.gz 2024-09-16T13:51:12.118 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.79936.log.gz 2024-09-16T13:51:12.118 INFO:teuthology.orchestra.run.smithi172.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.75128.log.gz 2024-09-16T13:51:12.118 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.97362.log.gz 2024-09-16T13:51:12.118 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44534.log.gz 2024-09-16T13:51:12.118 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34259.log.gz 2024-09-16T13:51:12.118 INFO:teuthology.orchestra.run.smithi172.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.85918.log.gz 2024-09-16T13:51:12.118 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster1-client.admin.36478.log.gz 2024-09-16T13:51:12.118 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.95236.log.gz 2024-09-16T13:51:12.118 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.92409.log.gz 2024-09-16T13:51:12.118 INFO:teuthology.orchestra.run.smithi172.stderr: 91.7% -- replaced with /var/log/ceph/cluster1-client.admin.49864.log.gz 2024-09-16T13:51:12.119 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.94336.log.gz 2024-09-16T13:51:12.119 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.85877.log.gz 2024-09-16T13:51:12.119 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.119813.log.gz 2024-09-16T13:51:12.119 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.100999.log.gz 2024-09-16T13:51:12.119 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.107536.log.gz 2024-09-16T13:51:12.119 INFO:teuthology.orchestra.run.smithi172.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.120647.log.gz 2024-09-16T13:51:12.119 INFO:teuthology.orchestra.run.smithi172.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.admin.37485.log.gz 2024-09-16T13:51:12.119 INFO:teuthology.orchestra.run.smithi172.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.admin.48992.log.gz 2024-09-16T13:51:12.119 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.67999.log.gz 2024-09-16T13:51:12.119 INFO:teuthology.orchestra.run.smithi172.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.86869.log.gz 2024-09-16T13:51:12.119 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41294.log.gz 2024-09-16T13:51:12.119 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53542.log.gz 2024-09-16T13:51:12.119 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.120155.log.gz 2024-09-16T13:51:12.120 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.138954.log.gz 2024-09-16T13:51:12.120 INFO:teuthology.orchestra.run.smithi172.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.admin.71270.log.gz 2024-09-16T13:51:12.120 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.62152.log.gz 2024-09-16T13:51:12.120 INFO:teuthology.orchestra.run.smithi172.stderr: 96.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61696.log.gz 2024-09-16T13:51:12.120 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41418.log.gz 2024-09-16T13:51:12.120 INFO:teuthology.orchestra.run.smithi172.stderr: 18.6% -- replaced with /var/log/ceph/cluster2-client.admin.49708.log.gz 2024-09-16T13:51:12.120 INFO:teuthology.orchestra.run.smithi172.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39422.log.gz 2024-09-16T13:51:12.120 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.89065.log.gz 90.0% 2024-09-16T13:51:12.120 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.32361.log.gz 2024-09-16T13:51:12.120 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54509.log.gz 2024-09-16T13:51:12.120 INFO:teuthology.orchestra.run.smithi172.stderr: 90.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56609.log.gz 2024-09-16T13:51:12.120 INFO:teuthology.orchestra.run.smithi172.stderr: 18.5% -- replaced with /var/log/ceph/cluster2-client.admin.142544.log.gz 2024-09-16T13:51:12.120 INFO:teuthology.orchestra.run.smithi172.stderr: 18.0% -- replaced with /var/log/ceph/cluster1-client.admin.75022.log.gz 2024-09-16T13:51:12.120 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.126792.log.gz 2024-09-16T13:51:12.121 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.80826.log.gz 2024-09-16T13:51:12.121 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.112627.log.gz 2024-09-16T13:51:12.121 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.30850.log.gz 2024-09-16T13:51:12.121 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79622.log.gz 2024-09-16T13:51:12.121 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.116898.log.gz 2024-09-16T13:51:12.346 INFO:teuthology.orchestra.run.smithi172.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.admin.36801.log.gz 2024-09-16T13:51:15.032 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33202.log.gz 2024-09-16T13:51:15.342 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79770.log.gz 2024-09-16T13:51:15.940 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.79562.log.gz 2024-09-16T13:51:16.931 INFO:teuthology.orchestra.run.smithi172.stderr: 89.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42643.log.gz 2024-09-16T13:51:22.269 INFO:teuthology.orchestra.run.smithi172.stderr: 91.5% -- replaced with /var/log/ceph/cluster2-client.admin.107949.log.gz 2024-09-16T13:51:22.269 INFO:teuthology.orchestra.run.smithi172.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.admin.49609.log.gz 90.0% 2024-09-16T13:51:22.269 INFO:teuthology.orchestra.run.smithi172.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.65056.log.gz 2024-09-16T13:51:24.104 INFO:teuthology.orchestra.run.smithi172.stderr: 90.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43905.log.gz 2024-09-16T13:52:19.974 INFO:teuthology.orchestra.run.smithi172.stderr: 93.6% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-09-16T13:57:44.876 INFO:teuthology.orchestra.run.smithi172.stderr: 96.2% -- replaced with /var/log/ceph/cluster2-client.mirror.27135.log.gz 2024-09-16T14:04:17.612 INFO:teuthology.orchestra.run.smithi018.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-09-16T14:05:19.350 INFO:teuthology.orchestra.run.smithi172.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-09-16T14:05:32.878 INFO:teuthology.orchestra.run.smithi018.stderr: 94.6% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-09-16T14:06:06.746 INFO:teuthology.orchestra.run.smithi018.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-09-16T14:06:06.747 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-09-16T14:06:06.748 INFO:teuthology.orchestra.run.smithi018.stderr:real 15m38.009s 2024-09-16T14:06:06.748 INFO:teuthology.orchestra.run.smithi018.stderr:user 10m39.572s 2024-09-16T14:06:06.748 INFO:teuthology.orchestra.run.smithi018.stderr:sys 0m34.390s 2024-09-16T14:13:46.544 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.61268.log.gz 2024-09-16T14:14:20.867 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.96362.log.gz 2024-09-16T14:14:30.546 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.64033.log.gz 2024-09-16T14:14:53.739 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.96368.log.gz 2024-09-16T14:15:04.480 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.49191.log.gz 2024-09-16T14:15:14.279 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.49197.log.gz 2024-09-16T14:15:16.516 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.37777.log.gz 2024-09-16T14:15:26.079 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.72092.log.gz 2024-09-16T14:15:26.792 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.119002.log.gz 2024-09-16T14:15:28.226 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.130511.log.gz 2024-09-16T14:15:49.364 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.130539.log.gz 2024-09-16T14:15:52.445 INFO:teuthology.orchestra.run.smithi172.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.admin.98066.log.gz 2024-09-16T14:15:52.578 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.85437.log.gz 2024-09-16T14:15:53.174 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.96350.log.gz 2024-09-16T14:15:55.300 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.85448.log.gz 2024-09-16T14:15:56.298 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.73198.log.gz 2024-09-16T14:15:59.623 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.52115.log.gz 2024-09-16T14:16:00.220 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.28135.log.gz 2024-09-16T14:16:02.992 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% 95.3% -- replaced with /var/log/ceph/cluster2-client.admin.107637.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.110259.log.gz 2024-09-16T14:16:02.992 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T14:16:04.226 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.108181.log.gz 2024-09-16T14:16:06.912 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.85138.log.gz 2024-09-16T14:16:08.727 INFO:teuthology.orchestra.run.smithi172.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.admin.51936.log.gz 2024-09-16T14:16:09.835 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.52287.log.gz 2024-09-16T14:16:11.239 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.37950.log.gz 2024-09-16T14:16:13.479 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.49267.log.gz 2024-09-16T14:16:13.504 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.62988.log.gz 2024-09-16T14:16:24.859 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.119204.log.gz 2024-09-16T14:16:24.859 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.108068.log.gz 2024-09-16T14:16:27.576 INFO:teuthology.orchestra.run.smithi172.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.admin.63855.log.gz 2024-09-16T14:16:28.447 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.49231.log.gz 2024-09-16T14:16:32.100 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.76170.log.gz 2024-09-16T14:16:32.719 INFO:teuthology.orchestra.run.smithi172.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.admin.98732.log.gz 2024-09-16T14:16:33.909 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.37679.log.gz 2024-09-16T14:16:34.374 INFO:teuthology.orchestra.run.smithi172.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.admin.75031.log.gz 2024-09-16T14:16:34.723 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.86922.log.gz 2024-09-16T14:16:40.409 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.120606.log.gz 2024-09-16T14:16:40.411 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.120743.log.gz 2024-09-16T14:16:40.412 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.118981.log.gz 2024-09-16T14:16:40.565 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.96354.log.gz 2024-09-16T14:16:42.468 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.99043.log.gz 2024-09-16T14:16:43.642 INFO:teuthology.orchestra.run.smithi172.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.admin.99513.log.gz 2024-09-16T14:16:44.531 INFO:teuthology.orchestra.run.smithi172.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.admin.52538.log.gz 2024-09-16T14:16:45.756 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.61117.log.gz 2024-09-16T14:16:47.337 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.39651.log.gz 2024-09-16T14:16:47.404 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.85179.log.gz 2024-09-16T14:16:47.997 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.74876.log.gz 2024-09-16T14:16:49.113 INFO:teuthology.orchestra.run.smithi172.stderr: 95.4% -- replaced with /var/log/ceph/cluster1-client.admin.40148.log.gz 2024-09-16T14:16:54.719 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.49209.log.gz 2024-09-16T14:16:54.903 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.119060.log.gz 2024-09-16T14:16:54.903 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.119059.log.gz 2024-09-16T14:16:55.613 INFO:teuthology.orchestra.run.smithi172.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.admin.111026.log.gz 2024-09-16T14:16:56.236 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.109636.log.gz 2024-09-16T14:16:57.548 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.110613.log.gz 2024-09-16T14:16:57.584 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.63464.log.gz -- replaced with /var/log/ceph/cluster1-client.admin.63154.log.gz 2024-09-16T14:16:57.584 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T14:16:58.184 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.96480.log.gz 2024-09-16T14:16:58.262 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.51182.log.gz 2024-09-16T14:16:58.584 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.75576.log.gz 2024-09-16T14:16:58.614 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.72093.log.gz 2024-09-16T14:16:59.686 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.61118.log.gz 2024-09-16T14:17:02.153 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.130478.log.gz 2024-09-16T14:17:02.996 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.142648.log.gz 2024-09-16T14:17:03.772 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.73147.log.gz 2024-09-16T14:17:04.062 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.28292.log.gz 2024-09-16T14:17:05.661 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.73193.log.gz 2024-09-16T14:17:06.345 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.119191.log.gz 2024-09-16T14:17:07.514 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.96461.log.gz 2024-09-16T14:17:08.475 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.85168.log.gz 2024-09-16T14:17:12.308 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.87366.log.gz 2024-09-16T14:17:12.387 INFO:teuthology.orchestra.run.smithi172.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.admin.63309.log.gz 2024-09-16T14:17:13.273 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.61132.log.gz 2024-09-16T14:17:13.808 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.86784.log.gz 2024-09-16T14:17:15.541 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.87210.log.gz 2024-09-16T14:17:15.568 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.49719.log.gz 2024-09-16T14:17:15.590 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.107575.log.gz 2024-09-16T14:17:15.594 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.108155.log.gz 2024-09-16T14:17:15.598 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.142639.log.gz 2024-09-16T14:17:17.986 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.142635.log.gz 2024-09-16T14:17:24.485 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.132819.log.gz 2024-09-16T14:17:24.487 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.119052.log.gz 2024-09-16T14:17:24.787 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.119010.log.gz 2024-09-16T14:17:27.283 INFO:teuthology.orchestra.run.smithi172.stderr: 95.4% -- replaced with /var/log/ceph/cluster1-client.admin.28781.log.gz 2024-09-16T14:17:28.463 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.37600.log.gz -- replaced with /var/log/ceph/cluster2-client.admin.37602.log.gz 2024-09-16T14:17:28.464 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T14:17:31.130 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.85133.log.gz 2024-09-16T14:17:32.399 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.107632.log.gz 2024-09-16T14:17:32.418 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.130488.log.gz 2024-09-16T14:17:33.554 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.110455.log.gz 2024-09-16T14:17:33.909 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.49273.log.gz 2024-09-16T14:17:34.088 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.49750.log.gz 2024-09-16T14:17:34.125 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.97927.log.gz 2024-09-16T14:17:34.582 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.96465.log.gz 2024-09-16T14:17:34.801 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.87056.log.gz 2024-09-16T14:17:35.431 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.27686.log.gz 2024-09-16T14:17:35.710 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.107741.log.gz 2024-09-16T14:17:38.802 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.96308.log.gz 2024-09-16T14:17:41.093 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.130995.log.gz 2024-09-16T14:17:41.093 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.130545.log.gz 2024-09-16T14:17:41.119 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.132486.log.gz 2024-09-16T14:17:42.176 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.96511.log.gz 2024-09-16T14:17:42.298 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.49181.log.gz 2024-09-16T14:17:42.785 INFO:teuthology.orchestra.run.smithi172.stderr: 95.1% -- replaced with /var/log/ceph/cluster1-client.admin.87541.log.gz 2024-09-16T14:17:42.897 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.39993.log.gz 2024-09-16T14:17:43.081 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.130928.log.gz 2024-09-16T14:17:43.448 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.37711.log.gz 2024-09-16T14:17:43.498 INFO:teuthology.orchestra.run.smithi172.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.admin.99335.log.gz 2024-09-16T14:17:43.626 INFO:teuthology.orchestra.run.smithi172.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.admin.121368.log.gz 2024-09-16T14:17:43.712 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.61228.log.gz 2024-09-16T14:17:43.956 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.39517.log.gz 2024-09-16T14:17:44.236 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.132963.log.gz 2024-09-16T14:17:44.277 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.40283.log.gz 2024-09-16T14:17:44.354 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.74741.log.gz 2024-09-16T14:17:45.095 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.108130.log.gz 2024-09-16T14:17:45.594 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.62854.log.gz 2024-09-16T14:17:46.285 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.27385.log.gz 2024-09-16T14:17:47.689 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.73153.log.gz 2024-09-16T14:17:49.460 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.87720.log.gz 2024-09-16T14:17:50.158 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.142576.log.gz 2024-09-16T14:17:51.879 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.98536.log.gz 2024-09-16T14:17:52.824 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.40704.log.gz 2024-09-16T14:17:53.267 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.142719.log.gz 2024-09-16T14:17:53.268 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.110085.log.gz 2024-09-16T14:17:53.271 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.119071.log.gz 2024-09-16T14:17:53.273 INFO:teuthology.orchestra.run.smithi172.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.admin.98219.log.gz 2024-09-16T14:17:53.710 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.73096.log.gz 2024-09-16T14:17:54.301 INFO:teuthology.orchestra.run.smithi172.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.admin.88093.log.gz 2024-09-16T14:17:54.480 INFO:teuthology.orchestra.run.smithi172.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.admin.98381.log.gz 2024-09-16T14:17:54.488 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.109770.log.gz 2024-09-16T14:17:54.794 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.62717.log.gz 2024-09-16T14:17:54.976 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.107579.log.gz 2024-09-16T14:17:55.323 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.133388.log.gz 2024-09-16T14:17:55.326 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.61229.log.gz 2024-09-16T14:17:55.487 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.37608.log.gz 2024-09-16T14:17:56.051 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.64331.log.gz 2024-09-16T14:17:56.537 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.130992.log.gz 2024-09-16T14:17:56.959 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.88289.log.gz 2024-09-16T14:17:57.517 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.142644.log.gz 2024-09-16T14:17:57.883 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.39247.log.gz 2024-09-16T14:17:59.401 INFO:teuthology.orchestra.run.smithi172.stderr: 95.4% -- replaced with /var/log/ceph/cluster1-client.admin.120879.log.gz 2024-09-16T14:17:59.477 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.133768.log.gz 2024-09-16T14:17:59.895 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.85131.log.gz 2024-09-16T14:18:01.330 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.133137.log.gz 2024-09-16T14:18:02.229 INFO:teuthology.orchestra.run.smithi172.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.admin.121033.log.gz 2024-09-16T14:18:02.695 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.130917.log.gz 2024-09-16T14:18:02.901 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.109949.log.gz 2024-09-16T14:18:02.942 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.27135.log.gz 2024-09-16T14:18:03.835 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.28604.log.gz 2024-09-16T14:18:03.857 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.109496.log.gz 2024-09-16T14:18:04.068 INFO:teuthology.orchestra.run.smithi172.stderr: 95.4% -- replaced with /var/log/ceph/cluster1-client.admin.27551.log.gz 2024-09-16T14:18:04.813 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.142733.log.gz 2024-09-16T14:18:04.933 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.107549.log.gz 2024-09-16T14:18:05.043 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.28448.log.gz 2024-09-16T14:18:05.133 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.49259.log.gz 2024-09-16T14:18:05.327 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.132622.log.gz 2024-09-16T14:18:05.730 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.85238.log.gz 2024-09-16T14:18:06.944 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.72056.log.gz 2024-09-16T14:18:07.488 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.73174.log.gz 2024-09-16T14:18:08.275 INFO:teuthology.orchestra.run.smithi172.stderr: 94.9% -- replaced with /var/log/ceph/cluster2-client.admin.73144.log.gz 2024-09-16T14:18:08.886 INFO:teuthology.orchestra.run.smithi172.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.admin.119176.log.gz 2024-09-16T14:18:09.592 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.85438.log.gz 2024-09-16T14:18:10.017 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.75362.log.gz 2024-09-16T14:18:10.251 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.39383.log.gz 2024-09-16T14:18:10.314 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.63639.log.gz 2024-09-16T14:18:11.206 INFO:teuthology.orchestra.run.smithi172.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.admin.74605.log.gz 2024-09-16T14:18:11.670 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.121993.log.gz 2024-09-16T14:18:11.893 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.75950.log.gz 2024-09-16T14:18:12.074 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.51626.log.gz 2024-09-16T14:18:12.217 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.51316.log.gz 2024-09-16T14:18:13.366 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.37739.log.gz 2024-09-16T14:18:14.449 INFO:teuthology.orchestra.run.smithi172.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.admin.51045.log.gz 2024-09-16T14:18:14.556 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.61125.log.gz 2024-09-16T14:18:14.669 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.134003.log.gz 2024-09-16T14:18:14.755 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.61330.log.gz 2024-09-16T14:18:15.086 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.39805.log.gz 2024-09-16T14:18:15.394 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.142911.log.gz 2024-09-16T14:18:16.004 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.51781.log.gz 2024-09-16T14:18:16.230 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.121189.log.gz 2024-09-16T14:18:16.823 INFO:teuthology.orchestra.run.smithi172.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.admin.75206.log.gz 2024-09-16T14:18:16.849 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.37601.log.gz 2024-09-16T14:18:17.229 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.110828.log.gz 2024-09-16T14:18:17.666 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.122190.log.gz 2024-09-16T14:18:19.369 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.130968.log.gz 2024-09-16T14:18:19.885 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.51451.log.gz 2024-09-16T14:18:20.497 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.133565.log.gz 2024-09-16T14:18:21.000 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.142578.log.gz 2024-09-16T14:18:21.881 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.37664.log.gz 2024-09-16T14:18:23.012 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.40458.log.gz 2024-09-16T14:18:23.325 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.61119.log.gz 2024-09-16T14:18:24.291 INFO:teuthology.orchestra.run.smithi172.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.admin.121603.log.gz 2024-09-16T14:18:25.130 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.27979.log.gz 2024-09-16T14:18:25.741 INFO:teuthology.orchestra.run.smithi172.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.admin.98889.log.gz 2024-09-16T14:18:25.857 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.27823.log.gz 2024-09-16T14:18:27.253 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.121780.log.gz 2024-09-16T14:18:29.124 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.85234.log.gz 2024-09-16T14:18:30.055 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.87916.log.gz 2024-09-16T14:18:30.441 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.142674.log.gz 2024-09-16T14:18:30.779 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.61212.log.gz 2024-09-16T14:18:31.797 INFO:teuthology.orchestra.run.smithi172.stderr: 94.8% -- replaced with /var/log/ceph/cluster2-client.admin.96317.log.gz 2024-09-16T14:18:37.965 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.132346.log.gz 2024-09-16T14:18:51.181 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.admin.75773.log.gz 2024-09-16T14:25:53.211 INFO:teuthology.orchestra.run.smithi172.stderr: 97.0% -- replaced with /var/log/ceph/cluster1-client.mirror.27164.log.gz 2024-09-16T14:34:11.087 INFO:teuthology.orchestra.run.smithi172.stderr: 95.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.27164.log.gz 2024-09-16T14:46:20.732 INFO:teuthology.orchestra.run.smithi172.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-09-16T14:47:40.132 INFO:teuthology.orchestra.run.smithi172.stderr: 95.5% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-09-16T14:49:14.173 INFO:teuthology.orchestra.run.smithi172.stderr: 95.3% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-09-16T14:49:14.174 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-09-16T14:49:14.174 INFO:teuthology.orchestra.run.smithi172.stderr:real 58m45.438s 2024-09-16T14:49:14.174 INFO:teuthology.orchestra.run.smithi172.stderr:user 40m53.040s 2024-09-16T14:49:14.175 INFO:teuthology.orchestra.run.smithi172.stderr:sys 2m25.009s 2024-09-16T14:49:14.176 INFO:tasks.ceph:Archiving logs... 2024-09-16T14:49:14.177 DEBUG:teuthology.misc:Transferring archived files from smithi018:/var/log/ceph to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907248/remote/smithi018/log 2024-09-16T14:49:14.178 DEBUG:teuthology.orchestra.run.smithi018:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T14:49:53.787 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/log/ceph to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907248/remote/smithi172/log 2024-09-16T14:49:53.788 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-16T14:52:24.893 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T14:52:24.905 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T14:52:24.905 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T14:52:24.908 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T14:52:24.990 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T14:52:24.998 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T14:52:24.999 DEBUG:teuthology.orchestra.run.smithi018:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T14:52:25.001 DEBUG:teuthology.orchestra.run.smithi172:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T14:52:25.098 INFO:teuthology.orchestra.run.smithi018.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T14:52:25.099 INFO:teuthology.orchestra.run.smithi018.stdout:============================================================================== 2024-09-16T14:52:25.099 INFO:teuthology.orchestra.run.smithi018.stdout:*hv01.front.sepi 67.205.162.81 3 u 21 256 377 0.176 3.729 0.413 2024-09-16T14:52:25.099 INFO:teuthology.orchestra.run.smithi018.stdout:+hv02.front.sepi 63.231.80.2 3 u 21 256 377 0.188 3.999 0.368 2024-09-16T14:52:25.099 INFO:teuthology.orchestra.run.smithi018.stdout:+hv03.front.sepi 74.6.168.72 3 u 130 256 377 0.199 3.351 0.549 2024-09-16T14:52:25.099 INFO:teuthology.orchestra.run.smithi018.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2024-09-16T14:52:25.118 INFO:teuthology.orchestra.run.smithi172.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T14:52:25.118 INFO:teuthology.orchestra.run.smithi172.stdout:============================================================================== 2024-09-16T14:52:25.119 INFO:teuthology.orchestra.run.smithi172.stdout:+hv01.front.sepi 67.205.162.81 3 u 89 128 377 0.166 0.802 3.086 2024-09-16T14:52:25.119 INFO:teuthology.orchestra.run.smithi172.stdout:*hv02.front.sepi 63.231.80.2 3 u 113 128 377 0.197 1.428 2.853 2024-09-16T14:52:25.119 INFO:teuthology.orchestra.run.smithi172.stdout:+hv03.front.sepi 74.6.168.72 3 u 119 128 377 0.197 2.269 2.271 2024-09-16T14:52:25.119 INFO:teuthology.orchestra.run.smithi172.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2024-09-16T14:52:25.120 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T14:52:25.129 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T14:52:25.129 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T14:52:25.139 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T14:52:25.149 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T14:52:25.161 INFO:teuthology.task.internal:Duration was 9584.534068 seconds 2024-09-16T14:52:25.162 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T14:52:25.178 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T14:52:25.178 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T14:52:25.181 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T14:52:25.445 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T14:52:25.445 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi018.front.sepia.ceph.com 2024-09-16T14:52:25.446 DEBUG:teuthology.orchestra.run.smithi018:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T14:52:25.474 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2024-09-16T14:52:25.474 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T14:52:25.503 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T14:52:25.503 DEBUG:teuthology.orchestra.run.smithi018:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T14:52:25.509 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T14:52:25.643 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T14:52:25.643 DEBUG:teuthology.orchestra.run.smithi018:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T14:52:25.648 DEBUG:teuthology.orchestra.run.smithi172:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T14:52:26.158 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T14:52:26.572 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T14:52:26.573 DEBUG:teuthology.orchestra.run.smithi018:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T14:52:26.618 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T14:52:26.660 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T14:52:26.789 DEBUG:teuthology.orchestra.run.smithi018:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T14:52:26.792 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T14:52:26.813 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2024-09-16T14:52:26.826 INFO:teuthology.orchestra.run.smithi018.stdout:kernel.core_pattern = core 2024-09-16T14:52:26.876 DEBUG:teuthology.orchestra.run.smithi018:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T14:52:26.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T14:52:26.882 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T14:52:26.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T14:52:26.889 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T14:52:26.900 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T14:52:26.902 DEBUG:teuthology.misc:Transferring archived files from smithi018:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907248/remote/smithi018 2024-09-16T14:52:26.903 DEBUG:teuthology.orchestra.run.smithi018:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T14:52:26.930 DEBUG:teuthology.misc:Transferring archived files from smithi172:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sangadi-2024-09-16_10:44:09-rbd-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7907248/remote/smithi172 2024-09-16T14:52:26.930 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T14:52:26.958 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T14:52:26.958 DEBUG:teuthology.orchestra.run.smithi018:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T14:52:26.968 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T14:52:26.974 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T14:52:26.986 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T14:52:26.987 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T14:52:26.999 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T14:52:27.000 DEBUG:teuthology.orchestra.run.smithi018:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T14:52:27.002 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T14:52:27.007 INFO:teuthology.orchestra.run.smithi018.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 14:52 /home/ubuntu/cephtest 2024-09-16T14:52:27.010 INFO:teuthology.orchestra.run.smithi172.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 14:52 /home/ubuntu/cephtest 2024-09-16T14:52:27.016 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T14:52:27.027 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T14:52:27.062 INFO:teuthology.run:Summary data: description: rbd/mirror/{base/install clients/{mirror-extra mirror} cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-hybrid supported-random-distro$/{ubuntu_latest} workloads/compare-mirror-images-nbd} duration: 9584.534067630768 failure_reason: '"1726489402.27115 mon.a (mon.0) 210 : cluster [WRN] Health check failed: Degraded data redundancy: 11/1270 objects degraded (0.866%), 1 pg degraded (PG_DEGRADED)" in cluster log' flavor: default owner: scheduled_sangadi@teuthology success: false 2024-09-16T14:52:27.062 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T14:52:27.184 INFO:teuthology.run:FAIL